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Abstract 


Let il C (N > 3) be an open domain which is not necessarily bounded. By using 
variational methods, we consider the following elliptic systems involving multiple 
Hardy-Sobolev critical exponents: 

f-Au- a '"' '^ = Kaj^\u\°‘-'^u\v\l^ inH, 

I -Av - ^ '"1 " = K/3j^\u\°‘\v\f^-^v inH, 


where Si, S 2 G (0, 2), a > 1, /3 > 1, A > 0, /i > 0, k 7 ^ 0, a -b /? < 2 *(s 2 ). Here, 
2*(s) := is the critical Hardy-Sobolev exponent. We mainly study the critical 

case (i.e., a + /3 = 2*( 52 )) when H is a cone (in particular, H or H = R^). 

We will establish a sequence of fundamental results including regularity, symmetry, 
existence and multiplicity, uniqueness and nonexistence, etc. In particular, the sharp 
constant and extremal functions to the following kind of double-variable inequalities 



+ 2*{s)k 


|u|“|u|^ 


)dx'j 


2 

2 ^ 


< [ (|Vup-b |Vup)dx 
Jn 


for {u, v) G ^ will be explored. Further results about the sharp constant Sa,p,\,^{Vl) 
with its extremal functions when O is a general open domain will be involved. 
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1 Introduction 


Let il C {N > 3) be an open domain which is not necessarily bounded. We study 
the following nonlinear elliptic systems 

f-Au- —- = Kaj^\u\°‘~'^u\v\^ infl, 

I = kI3j^\u\‘^\v\^-^v inn, (1-1) 

[(u,u)e^:= Dl’\il)xDl’^{n), 

where si, S 2 S (0,2), a > 1, /3 > 1, A > 0, /i > 0, k ^ 0, a+/3 < 2*(s2) := 

The interest in studying the nonlinear Schrodinger systems is motivated by real 
problems in nonlinear optics, plasma physics, condensed matter physics, etc. For ex¬ 
ample, the coupled nonlinear Schrodinger systems arise in the description of several 
physical phenomena such as the propagation of pulses in birefringent optical hbers and 
Kerr-like photorefractive media, see ||2] [TTl [16] |23| |24| |28l, etc. The problem comes 
from the physical phenomenon with a clear practical significance. The researches on 
solutions under different situations not only corresponds to different physical interpre¬ 
tation, but also has a pure mathematical theoretical significance. Hence, the coupled 
nonlinear Schrodinger systems are widely studied in recently years, we refer the read¬ 
ers to ||T]|3]|2^IlllEl] and the references therein. 

For any s € [0, 2], we dehne the measure d/ig := and ||u||^ j, := / \u\^dpLg. 

' Jq 

We also use the notation ||u||p := ||u||p,o- The Hardy-Sobolev inequality llsl l7l [141 
asserts that ^ (R^, d/ig) is a continuous embedding for s G [0,2]. 

For a general open domain H, there exists a positive constant C{s, H) such that 

iVupdx > C{s, ^ ^ w 

Dehne /Xg^ (H) as 


Mg,(H) :=iuf{ , uGZ?1’2(H)\{0}}. (1.2) 

Consider the case of H = it is well known that the extremal function of /ig^ (R+) 
is parallel to the ground state solution of the following problem: 


|.|n 

■u = 0 on 


in 




(1.3) 


We note that the existence of ground state solution of (11.3b for 0 < si < 2 is solved 

by Ghoussoub and Robert |[T3l . They also gave some properties about the regularity, 

_ ^-2 

symmetry and decay estimates. The instanton U{x) := C[k + for 
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0 < S 2 < 2 is a ground state solution to Ol) below (see Ha and ETl): 


IM > 0 in and 



(1.4) 


u —>■ 0 as |x| —>■ + 00 . 


The case that 0 € dfl has become an interesting topic in recent years since the curvature 
of dfl at 0 plays an important role, see ID[JJ][SIUS), etc. A lot of sufficient conditions 
are given in order to ensure that in those papers. It is standard 

to apply the blow-up analysis to show that (O) can be achieved by some positive 
u £ (52) (e.g., see lIT^ Corollary 3.2]), which is a ground state solution of 



in 52, 
on 952, 



However, it seems there is no article before involving the system case like (II.Il l with 
Hardy-Sobolev critical exponents, which we are going to deal with in the current paper. 
It is well known that the main difficulty is the lack of compactness inherent in these 
problems involving Hardy-Sobolev critical exponents. The compactness concentration 
argument (see im, etc.) is a powerful tool to handle with these critical problems. It 
is also well known that the compactness concentration argument depends heavily on 
the limit problem. Consider a bounded domain 52, if 0 ^ 52, we see that * = 1,2 
are regular. We are interested in the case of that 0 £ 52. It is easy to see that when 
0 £ 52, the limit domain is and when 0 £ 952, the limit domain is usually a cone. 
Especially, when 952 possesses a suitable regularity (e.g. 952 £ at a: = 0), the limit 
domain is after a suitable rotation. Hence, in present paper, we mainly study the 
critical elliptic systems (fTTT i with Of -f /3 = 2* ( 52 ) and 52 is a cone. 

Definition 1.1. A cone in is an open domain 52 with Lipschitz boundary and such 
that tx £ VLfor every 2 > 0 and a: £ 52. 

We will establish a sequence of fundamental results to the system ( II.lb including 
regularity, symmetry, existence and multiplicity, and nonexistence, etc. Since there are 
a large number of conclusions in the current paper, we do not intend to list them here. 
This paper is organized as follows: 

In Section 2, we will establish by a direct method a type of interpolation inequalities, 
which are essentially the variant Caffarelli-Kohn-Nirenberg (CKN) inequalities, see 


0 . 


In Section 3, we will study the regularity, symmetry and decay estimation about the 
nonnegative solutions of (fTTT i. Taking as a specific example, we will study the 
regularity based on the technique of Moser’s iteration (see Proposition 13.1b . By the 
method of moving planes, we obtain the symmetry result (see Proposition |33]l. Due to 
the Kelvin transformation, we get the decay estimation (see Proposition l3.2b . 
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In Section 4, we shall study the basic properties of the corresponding Nehari manifold. 


In Section 5, we will give a nonexistence of nontrivial ground state solution of (11.11 
for the case S 2 > si, see Theorem 15. II 


In Section 6, we will give an existence result of positive solution result for a special 
case : A = ^ , see Corollary 16. II Further, we prepare a sequence of pre¬ 

liminaries for the existence result which are not only useful for us to study the case of 
Si = S 2 in Section 7, but also the case of si ^ S 2 in Section 8. 


In Section 7, we will focus on the case of si = S 2 = s € (0, 2) when is a cone. In 
this case, the nonlinearities are homogeneous which enable us to define the following 
constant 


A,/i(Al) .— inf ^ 


In + |Vu|2)dx 




_L _L 2*(c)- 


|a:p 


M- 






jdxj 


(1.5) 


where 


|y|2*(s) |^|2*(«) 


:={(«,«)£®: y^(A-j^+M^p + 2-(.) 


'‘ "IJJ )<ii>0}. (1.6) 


In particular, we shall see that ^ = ^\{(0, 0)} if and only if 


a p 


see Lemma I tTtI When k < 0, we will prove that has no nontrivial ex¬ 

tremals (see Lemma Il3. Hence, we will mainly focus on the case of k > 0 and 
show that the system (O possesses a least energy solution and that is 

achieved. These conclusions will produce the sharp constant and extremal functions to 
the following kind of inequalities with double-variable 


a luP I 


.|2*(6i) 


-f 2 *(s)k 


\ur\v 


1^ 




2^ 


< [ {\\7u\^ + \Vv\^)dx 

Jn 

for {u, v) € 3). For this purpose, a kind of Pohozaev identity will be established. Then 
the existence, regularity, uniqueness and nonexistence results of the positive ground 
state solution to the system ( 11.11 ) can be seen in this section. Under some proper hy¬ 
potheses, we will show that the positive ground state solution must be of the form 
[C{to)U,toC{to)U), where to > 0 and C{to) can be formulated explicitly and U is 
the ground state solution of 

(-Au = psin)^ 

]u = 0 
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Taking a special case = 3, s = 1, a = /3 = 2, A = p = 2 k in consideration, we will 
find out all the positive ground state solutions to (fTTT) . Based on these conclusions, 
we may prove the existence of infinitely many sign-changing solutions of the system 
(O on a cone fl by gluing together suitable signed solutions corresponding to each 
sub-cone. Further, if O is a general open domain, the sharp constant and 

its extremal functions will be investigated. We will find a way to compute 
and to judge when can be achieved if fl is a general open domain. 

In Section 8, the system dl.lb satisfying si S 2 G (0, 2) will be studied. We shall 
consider a new approximation to the original system (O- The estimation on the least 
energy and the positive ground state along with its geometric structure to the approx¬ 
imation will be established. Finally, the existence of positive ground state solution to 
the original system will be given. 


2 Interpolation inequalities 

For Si 7^ S2, we note that there is no embedding relationship between d^si ) 

and d^s^) for domain fl with 0 G fl. Flence, we are going to establish 

some interpolation inequalities in this section. 

Proposition 2.1. Let fl C K.^(iV > 3) be an open set. Assume that 0 < si < S 2 < 
S 3 < 2, then there exists 9 = ^ S (0,1) such that 

— (iV-S2)(s3-Sl) ' ’ ' 

I“|2 *(s2),*2 < I“I2*(si)..J^I2m!3),.3 (2 -1) 

for all u e L2*(*i)(f2, n ^). 

Proof. Define p = , then 1 — n = . A direct calculation shows that 

S2 = psi + (1 - p)s3 (2.2) 


and 

2*(s2) = p2*(si) + (l-p)2*(s3). 

It follows from the Holder inequality that 


|2*(^<2) 


In Fl' 


-dx = 


< 




’ ^ a; 


n F 


Si 


-dx 


|2*(s 3) \ 1-e 

;—;- dx] 


n FI' 


(2.3) 


Let 9 := g, then by (12.31) again, \ — 9 = — q). Then we obtain that 

|w|2*(s2).S2 ^ I'“l2*(si).si |r''l2*(s3),S3 
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for all M e 


dx 


), where 


2*(si) _ {N - Si)(53 

2 *( s 2 )^ {N - S2){s3 


S 2 ) 

Si) 


e(o,i) 


has the following properties. Firstly, we note that 9 > 0 since si < S 2 < S 3 < 2 < N. 
Secondly, 


9 <1^ {N - Si)(s3 - S2) < {N - S2)(S3 - Si) (S2 - Si)( 7 V - S3) > 0 . 


□ 

Define 

i?(si, S 2 ) := ^ for 0 < Si < S 2 < 2. (2.4) 

S2(iV - Si) 

Corollary 2.1. Let LI C ]R^(iV > 3) be an open set. Assume 0 < si < 2. Then for 
any S 2 € [si, 2] and 9 G [t5(si, S 2 ), 1], there exists C{9) > 0 such that 

|w|2.R).«i <^(^)IKII%l2:'(t)... (2-5) 

for all u G Dg’^(r2), where ||m|| := (jVMpdx) 

Proof If S 2 = Si = s, then i?(si, S 2 ) = 0 and (12.5b is a direct conclusion of Hardy- 
Sobolev inequality and the best constant C{9) = ps{Ll)~^ y 9 G [0,1], where p.s{^) 
is defined by (11.2b . If si = 0, then t9(si, S 2 ) = 1,0=1 and (12.5b is just the well- 
known Sobolev inequality. 

Next, we assume that 0 < si < S 2 < 2. We also note that if 0 = 1, (12.5b is just the 
well-known Sobolev inequality. Hence, next we always assume that 0 < 1. Define 

- _ {N - S2)(S2 - Si) 

0(iV - Si) - (S2 - Si) ■ 

Note 0 G ['0(si, S 2 ), 1), we have that 0 < s < si < S 2 < 2. Then by Proposition 12.11 
we have 

l'*^|2*(si),si < l“l2*(s).sl'*^l2*(s2).S2' 

Recalling the Hardy-Sobolev inequality, we have 

Hence, there exists a (7(0) >0 such that 

\n\2HsA,s^<Cm\n\fK4^),s.. 

□ 

Define 

c(si, S 2 ) := ^^ for 0 < Si < S 2 < 2. (2.6) 

(N - S2)[2 - si) 
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Corollary 2.2. Let O C ]R^(7V > 3) be an open set. Assume 0 < S 2 < 2. Then for 
any si G [0, S2] and a G [0, S2)], there exists a C(a) > 0 such that 


|m|2*(s 2).S2 < C'(o-)l|u|r ‘^|m|2.(si),si (2-7) 

for all u G Dg’^(fl). 

Proof We only need to consider that case of si < S 2 and cr > 0. Define 

_ {N - Sl)(s 2 - Si) 

{N - Si) - {N - S2)<j' 

Recall that a G (0, ^(si, S2)], we have 0 < si < S2 < s < 2. Then by Proposition l2.ll 
we have 

I“|2 *(s2 ).S2 ^ I“l2*(si),si I“l2*(s),s' 

Recalling the Hardy-Sobolev inequality, we have |m| 2 *(s),s < llt^ll- Hence, 

there exists a C{a) > 0 such that 


I“|2*(s 2).S2 <C'(o-)l|u|r ''|w|2*(si).si- 

□ 

Remark 2.1. The above Corollarv \2.1\ and Corollarv \2.2\ are essentially the well known 
CKN inequality. However, based on the ProDosition \2.1\ our proofs are very concise. 
Moreover, the expressions of dO and dO l are very convenient in our applications. 


3 Regularity, symmetry and decay estimation 


In this section, we will study the regularity, symmetry and decay estimation about the 
positive solutions. 

Lemma 3.1. Assume that 0 < si < S 2 < 2,0 < u G and\u\‘^ G 

for all 1 < g < gi, where := i?i(0) fl R;!^. Then 

|„|-(->-/l«l» C for all 1 < , < jv(Ar ^-1!)"+fa ' 

Further, if\u\'^ G L^{B^) for all 1 < g < oo, then we have 

|up*("=)-7|a;|"" € L<^{Bt) for all l<q< 

[N + 2 )(s 2 - si) 


Proof. When q < 


_tV(A^+ 2 — 2 si _ 

A^(tV+2 —2 s2) + (A^+2){s 2 )gi 


and 0 < Si < S 2 < 2, we see that 


2*72) - 1 g ^ ^ ^ ^ 


2*(si) - 1 gi 


N 









Then we can take some 6 G (0,1) such that 


2^ 

2*(si) 


ll. 

1 qi 


<9<1 


S 2 q 


2*(s2)-1 


N 


Let 

, - 12 *(s 2)-1 

1-61 ’ 6» 2*(si) - l*^' 

Then by the choice of 9, we have t < N and q < qi. Hence, by the Holder inequality, 
we have 



y(2*(s2)-l)« 


dx < 



(2*(si)-l)(j ^0 

-—j —=—dx 

xSiq 


'+ X 


1 -e 


< + 00 . 


(3.1) 


, . iV(iV + 2-2si)qi 

It IS easy to see that — 7 ——-—-—^- 7 - 7—-77 -^— 

N (^N + 2 — 2 S 2 ) + {N + 2)(s 2 — 'Si)(7i 

N(N + 2-2si) 


is increasing by qi and 
□ 


Lemma 3.2. Assume that 0 < S 2 < si < 2, 0 < u € and\u\‘^ S 

L‘‘) for all 1 < q < qi, where := i?i(0) fl Then \u\^ £ 

L‘>{Bf)forall l<q< |r|§y^9i. 

Proof For any 1 < g < we set t = - S 2 q and q = l,[ll]z\ q- 

Then under the assumptions, it is easy to see that f > 0 and 1 < q < qi. Hence, 


u 


(2*(s2)-1)<j 


U, S29 


-dx = 


u 


(2*(si)-1)9 


< 


L 


.+ \xf^t 

u(2*(si)-1)9 
.+ |a;|®i9 


\xydx 
dx < +c». 


(3.2) 

□ 


We note that for some subset Hi and some g > 1 such that 

then by Holder inequality, we also have ^ £ L'^(Hi) provided 0 < tl^t2 < 

2 *(s 2 ) — 1 and fi + 62 = 2 *(s 2 ) — 1. Hence, we can obtain the following result: 

Proposition 3.1. Assume si, S 2 £ (0,2), k > 0, a > 1, /3 > 1, a + /3 = 2*(s2), then 
any positive solution (u, v) of 

-Am - —- = Kaj^\u\°'~'^u\vf inR^, 

* -Am-p '"' " = m/3|^|m|°|m|^~^m inR ^, (3.3) 

Jm,m)£^:= dI’\R^) X dI’\R^), 
satisfying the following properties: 
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(i) ifO < max{si,S 2 } < then u,v & C‘^{R^); 

(ii) i/maxjsi, S 2 } = thenu,v £ for all Q < 7 < 1 ; 

{in) !/max{si, S 2 } > thenu,v G C^’’^{R^)for all 0 < 7 < 

Proof. Indeed, it is enough to consider the regularity theorem at 0 £ dM.^. By Il26l 
Lemma B.3], u,v are locally bounded. Let := Bi{0) fl We see that there 
exists some C > 0 such that 

|u(a;)p*("=)-V|a;|*^ < C\x\-^f\v{x)f‘'^^'^-^/\xf^ < C\x\-^^ (3.4) 

for X £ B^. Hence 

e L‘i{B+) for all 1 < g < — 

Si 

and 

Ka-^\u\°‘-‘^u\v\^ , kI3-^\u\°‘\v\^-‘^v £ L«(B+) for all 1 < g < —. 

Set Smax '■= max{si,S 2 } and Smin '■= min{si,S 2 }- Then we have that u,v € 
W^’‘>{B+) for all 1 < g < Denote 

Tu := sup{t : sup(|u(x)|/|a;|’') < 00 , 0 < r < 1}, 

Bt 

Tv := sup{t : sup(|u(a;)|/|a:|'^) < 00 , 0 < r < 1}. 

Bt 

and 

To := min{T„,r„}. 

Step 1: We prove that tq = 1, i.e., Tu = Tv = 1. 

Case 1: Smax < 1- By the Sobolev embedding, we have u,v G C'^{B^) for any 
0 < r < 1. Hence, tq = 1 in this case. 

Case 2: Smax > !• For this case, we have u, u £ ) for all 0 < r < min{2 — 

Smax, !}■ Then by the definition, we have 2 — Smax < "ni < 1- For any 0 < r < tq, 
we have |it(a;)| < C\xf and |u(x)| < C\xf for x £ B^, then for any x £ B^, there 
exists some C > 0 such that 

< C|a;| 

\u{x)f^^^'^-^/\x\^^ < 

10 


(3.5) 






Suppose To < 1, then by (13.5b . there must hold {2*{smax) — l)r'o — Smax < 0. 
Otherwise, 

|y|2*(Smax)-l^|2;|Smax | ^ | 2* (^max ) - 1 ^ | | S^ax g 


for all 1 < <7 < oo. On the other hand, by Lemmaand Holder inequality, it is easy 
to prove that 

Ka--^—\u\‘^~'^u\vf, kP \u\‘^\vf~'^v G for all 1 < g < oo. 

\x\ 2 |a;| 2 

It follows that u G (^ 1 / 2 ) 1 < g < 00 and then by the Sobolev embedding 

again we have tq = 1, a contradiction. Therefore, (2*{smax) — 1)to — Smax < 0 is 
proved and thus we have 

|up*(®”‘“x)-l/|2.|Smox^ |^|2*(Sm„x)-l^|2.|Smax g L? (ij+) 


for all 1 < g < - 

^max 


N 

(2*(Smax) - 1 )t-0 


Subcase 2.1: If Smin < 1 or (2*{smin) ~ l)Tb ~ ^ 0, we have 

|^;| 2 *(«min)-ly'|^|Smin g (- 75 +) 

for all 1 < g < TV. We claim that Smax — {‘^*{smax) — l)'T) > 1- If not, we see 
that u,v G for all 1 < q < N, and then by Sobolev embedding, we 

obtain that tq = 1, a contradiction. Hence, we have it, u G ) for all 1 <q< 

-7 —— -T—, and by the Sobolev embedding again, we have it, u G i^i/ 2 ) 

Smax — ^2* (Smaa:) — 1 J Tq 

for all 0 < r < min{2 — [smax — (2*(Smax) — l)'T)] i !}■ Then by the dehnition of 
To, we should have 

2 [^TTiaa; (2 {Sjnax) f)^o] ^ '^O 

which implies that 

2 Smax “t” (2 {Smax) 2^To ^ 0. 

But Smax < 2, 2*(smax) > 2, Tq > 0, a Contradiction again. 

Subcase 2.2. If 1 < Smin Pi Smax ^ 2 and ^2 (.Smin) l)ro Smin ^ O 5 f^y Lemma 
I3.2l again. 

|lt| 2 *('’”‘’")“l/|a;|®'"‘”, g L‘> {B^) 

for all 

, ^ 2 *{Smax) - 1 N 

1 < 9 < -7-^- 7 -N-■ 

2 [Smin) 1 Smax (2*(Smax) Ij't'O 

On the other hand, by the dehnition of tq, we have that 
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for all 1 < <7 < 
\u 

for all 

1 < <7 < max I 
Noting that 


N 

2 (Smin) 1 

N 


. Thus, 

|^|2*(Smin)-ly'|2.|Smin g L‘> ) 


2 *{Sma.) - 1 


N 


^min l)'to ^ (^min) 1 ^max {p'*{^max^ l)^)- 


N 


2*(Sma^) 

- 1 

N 



1 Smax 

(2* {Smax^ 

- 1 ) 7-0 


N 



^max 


- 1)7-0 



N 



Smin ~ 


- 1 ) 7 - 0 ’ 



|^|2 {Srnin)- 

-1 ! \^\Srr,ir, 

G L«(B+) for all 1 < g < 


and it follows that 


X ^ 2 * (Smin) "^0 

u,v G for 1 < g < 


N 


Smax (^max) 1)^0 


Then apply the similar arguments as that in the subcase 2.1, we can deduce a contra¬ 
diction. Hence, tq = 1 is proved and then = t„ = 1, i.e., for any 0 < r < 1, 


< C|:e| 


— ilT —Si 


-i IT-Si 


-i Jr-S2 


-i Jr-S2 


(3.6) 


Step 2: We prove that u,v G W^’'^(B^) for all 

.. J tX) if 2 {s^ax') 1 ^max ^ 0 

— ^ ^ ) _ N _ if 9 * to i_ 1 _c 

1 1 c _9*/'c 't ^ \^max J ^ ^max ^ 

V, -*- I *max ^ \^max) 

We divide the proof in two cases. 

Case 1: 2*{smax) - 1 - Smax > 0, i.e., Smax < By taking r close to 1, we see 

that 
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for all I < q < oo. Meanwhile, by the Holder inequality, 


Ka -.—; \U\ 

a; ' ' 


-^u\v\^, G L«(5+) 


for all 1 < q < oo. 


Hence, u,v G for all 1 < g < cxd. 

2 

Case 2: 2*{smax) - 1 - Smax < 0, i.e., < Smax < 2. In this case, we have 


for all 




1 < g < 


,|2 (Smax) 1 /It-I®' 




N 


1 + ^max 2 (SjTjax) 

lf2^{Srmn)-l-S min ^ 0 , then we see that 
|u|2*('*™‘")“i/|a;|®™*", |^|2*(smin)-iy'|^|Smin g L'^(i?+) for all 1 < g < OO. 

Hence, u,v G for all 1 < g < rr—--r. 

’ Vi/ — Vf l-\-Smax—2*{Smax) 

If 2*{smin) - 1 - Smin < 0, we must have 

| 2 *(®’">")“i/|x|G L'^{B^) 


\y^\2* (Smix.) l/|a;|®" 


for all 1 < g < 


_ N 

IG^Tnin (^min) 

N 


. Since 


> 


N 


1 + 5mm 2 (Smm) “ 1 + Smax 2 (Smaa:) 

we also obtain that u^v G (Bt ) for all 1 < g < j— - ^^^^7 -r. 

^ ^~rSmax ^ 

Step 3: By the Sobolev embedding theorem, 

iV + 2 


i,vG C'^’'^(S+/ 2 ) for all 0 < 7 < 1 if Smax ^ 


N 


In particular, in the case Smax < there exists qo > N such that 

11^11^3.< 10 ( 5 + 2 ) 

X2*(®i)-2VU 


+ |2.|si ll5<J0(B+) + II |2.|si + l IIloo(B+) + II |^|s2 H 


“ '^v^S/u, 


S2 llL'>o(B+) 


|U“ V u II ^ II II 

I ll5oo(B+) + II |2 .|s2 + 1 \\loo(B+)} 


< OO. 


Thus, we obtain that u G C^{B^^.^). Similarly, we can also prove that v G C‘^{B^^.^). 


If Smax > note that -r > N, by taking r close to 1 , we have 

/»i + Smax—^ ySinax) 


U,VG C'^’+(B+/2) for all 0 < 7 < 1 - [1 + Smax - 2*{Smax)] = 


-^(2 -Smax) 


N-2 


□ 
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Proposition 3.2. Assume that si,S 2 S (0, 2), k > 0, a > 1, /3 > 1, a + /3 = 2*(s2)- 
Let (m, v) be a positive solution of (El, then there exists a constant C such that 


k(a;)| < C(1 + |a;| |VM(a;)|, |Vi;(a;)| < ( 710:1 

Proof. Recalling the Kelvin transformation: 

u*{y) ■■= v*(y) := (3.7) 

It is well known that 

and Av*{y) = ^^(Ar;)(^). (3.8) 


Hence, a direct computation shows that (it*, u*) is also a positive solution to the same 
equation. 

By Proposition 13.11 we see that u*,v* S (7^’'>'(]R([f) for some 7 > 0. Then 
|'«*(i/)|, |u*(2/)| < C\y \ for y G B^. Going back to (u, v), we see that \u{y)\, |f( 2 /)| < 
for y G R;?^. Finally, it is standard to apply the gradient estimate, we 
obtain that |Vu(2/)|, |Vi;(2/)| < C\y\~^ for y G R+. □ 


Remark 3.1. Checking the proofs i 
conclusions are valid for general cone LI. A little difference is that when LI = 
decay estimation is 


|ii(a;)|, |i;(x)| < (7(1 + |x| ^); |Vtt(a:)|, |Vi;(x)| < (7|x| ^ \ 


Proposition 3.3. Assume that si, S 2 & (0, 2), k > 0, a > 1, /? > 1, a + /3 = 2*(s2). 
Let (it, v) be a positive solution of (El. Then we have that u o a = u,v o a = v 
for all isometry o/R^ such that (^(R^j^) = M.^. In particular, {nix', xn), v{x' , xn)) 
is axially symmetric with respect to the XN—axis, i.e., u{x',xn) = u{\x'\,xn) and 
v(x',Xn) = v(\x'\,xn). 

Proof. We prove the result by the well-known method of moving planes. Denote by 
e)v the vector of the canonical basis of R^ and consider the open ball D := 
Set 

hix):=\xr^u{-^+j^), 

\f){x):=\x\'^-^v{-e]^ + j^) 

for all X G i7\{0} and (/^(O) = 'ijj{0) = 0. By Proposition l3.ll p{x), fjix) G C‘^[D) n 
(7^(i)\{0}). We note that it is easy to see that p{x) > 0, ip{x) > 0 in Z? and p{x) = 
'0(a;) = 0 on c)£)\{0}. On the other hand, by Proposition l3.2l there exists (7 > 0 such 
that 

p{x) < C\x\,'ij}{x) < C\x\ for all x G Z)\{0}. (3.10) 
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Since (/j( 0) = '0(0) = 0, we have that 0 (x) S C^{D). By a direct computation, 
{}p{x)^ 0 ( 2 ;)) satisfies the following eqaution 


Noting that 
we have 


—— A 
—A0 — /r 


a:—|a:P^| 


Ka 

k/3 


a: —la^Pejv | 

a:—|a:peiv 


in D. 


X — |a;peJv| = ItcNa; — e/v|, 


— A(^ — At 




^T] : 

ai —e 

. o* / = . \ 


-A0- 


I 


■^ = na-^ 

= K0 t 


x — e^ 


in D. 


(3.11) 


(3.12) 


(3.13) 


Since eJv G 9Z?\{0} and ip{x), 0(a;) G C'^(i)\{0}) nC'°(i)), there exists C > 0 such 
that 

V 3 (a;) < C\x — e)v|, 0(a;) < C\x — e/v| for all x £ D. (3.14) 


Noting that 2*{si) — 1 — Si > —N for i = 1,2, then by (13.101) . (13.131) . (13.14b and the 
standard elliptic theory, we obtain that (p{x), ip{x) £ {D). By (p{x) > 0,0(a;) > 0 

in D, we obtain that ^ < 0, ^ < 0 on dD, where v denotes the outward unit normal 
to D at X G dD. 

For any p > 0 and any x = (xi, x') G where x' = (x 2 , • • • ,xn) £ 
we let 

Xr/ = (2rj — xi,x) and := {x G £l|x^ G D}. (3.15) 

We say that (P^) holds iff 


7 ^ 0 and (/ 9 (x) > (^(x^), 0 (x) > 0 (x^) for all x £ such that xi < rj. 


Step 1: We shall prove that (P^) holds if 77 < 2 and close to 2 sufficiently. 

Indeed, it is easily to follow the Hopf’s Lemma (see the arguments above) that there 
exists £0 > 0 such that (P^) holds for rj £ (2 — £0, ^). Now, we let 

cr := minlp > 0; (Ps) holds for all <5 G ( 77 , i)}. (3.16) 


Step 2: We shall prove that cr = 0. 

We prove it by way of negation. Assume that cr > 0, then we see that ^ 0 and that 
(Per) holds. Now, we set 

0(x) := (/ 5 (x) - '^{Xa) and '0(x) := 0(x) - ip{xcr). (3.17) 
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Then we have 


-A(p =[-A(/?(a;)] - [-A(/?(a;^)] 


=A 


{ip{x)) 


2*(si)-l 


\x - 


Ka 


-X 




2*(si)-l 


{ip{x)y \^{x)y 

\x - 

{(p{Xa))°' ^{tp{x^))^ 


\x^-\x,^^ \x^ - \x,\^F^\^- 

2*(si)-lr 




la; - |a;Pi}^|si la;^ - b^pewh 


+ Ka{ip{Xa))°‘ ^{tlj{x))^[ 


\x-\xPeN\^^ la;^ - b^Pe^vh' 


for all X G Da H {xi < a}. Noting that 


I I i2—I I i2— 

|a;cr + |a;cr| cn] - |a; + |a;| eAr| 

= (|2;o-|^ — |a;|^)(l + |xo-|^ + |a;|^ + 2 xn) 
= 4a(a - a;i)(l + \xa\'^ + \x\^ + 2 xn), 


we obtain that 


— A(f{x) > 0 for all x G Da H {xi < a}. 
Similarly, we also have 

— A'^{x) > 0 for all x G Da fl {a;i < cr}. 


(3.18) 


(3.19) 

(3.20) 

(3.21) 


Then by the Hopf’s Lemma and the strong comparison principle, we have 

(^, V; > 0 in Da D {xi < a} and ^ < 0 on Da H {xi = cr}. (3.22) 

au ov 

Here we use the assumption ct > 0. By definition, there exists a subsequence {(TijigN C 
R+ and a sequence C D such that at < a,x^ G Da^ ,{x^)i < (Ji, lim ai = a 

i—^oo 

and 

< (p{{x'')ai) or V'(x*) < V'((x*)^J. (3.23) 

Up to a subsequence, we may assume that (p{x^) < (p{{x'^)ai) without loss of gen¬ 
erality. Since {x®}igN is bounded, going to a subsequence again, we assume that 
lim Xi = X G Da n {xi < cr} due to the choice of {x*}. Then we have (p{x) < 

i—^OO 

(p{xa), i.e., ifix) < 0. Combining with (13.23b . we obtain that ip{x) = 0 and then 
X G d{Da n {xi < cr}). 

Case 1: If X G dD, then f{x) = 0. It follows that ipixa) = 0. Since Xa G D and 
(fi > 0 in 79, we also have Xa G dD. We say that x = Xa- If not, x and Xa are 
symmetric with respect to the hyperplane xi = cr. This is impossible since that 79 is a 
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ball, a > 0 and x, Xa S dD. Now recalling that (f € C^, by the mean value theorem, 
there exists a sequence Ti G ((x®)i, 2 ai — (x*)i) such that 

ip(x") - ip{{x")^.) = 2di(p{Ti, {x'Y) ((x*)i - CTi). (3.24) 


Using the facts (x*)i < (Ti and (p{x'') < we let i go to infinity and then 

obtain that 


di(p{x) > 0 . 


(3.25) 


On the other hand. 


di(p{x) 


a d(p 
\x — icTvl dv 


< 0 , 


(3.26) 


a contradiction. Here we use the assumption cr > 0 again and the fact xi = cr since 

X = Xa- 


Case 2: If X € D, since (p{x) = (p{xa) > 0, we have Xa G D. Since x G d[Da D 
{xi < cr}), we have x G D r\ {xi = cr}. Then apply the similar arguments in 
Case 1, we can also obtain that di(p{x) > 0. On the other hand, by (13.231) . we have 
2 di(p{x) = di(p{x) < 0, also a contradiction. 

Hence, cr = 0 is proved. 

Step 3: By Step 2, cr = 0. Hence, we have 

<p(xi,x') > xi,x'), V^(xi,x') > '!/'(—xi,x') for all x G Dq = D. (3.27) 

Apply the same argument one can obtain the reverse inequality. Thus, 

(p{xi,x') = (p{—xi,x'),ilj{xi,x') = '!/'(—xi,x') for all x G D. (3.28) 

Hence, tp and ip are symmetric with respect to the hyperplane {xi = 0}. Noting that 
the arguments above are valid for any hyperplane containing ejv- By going back to 
{u, v), we obtain the conclusions of this proposition. □ 

Remark 3.2. (Open problem) The Pror)osition \3.3\ is established under the assump¬ 
tion that n = But so far we do not know whether the Proposition I3.5I is true for 
general cone H. It remains an open problem. 


4 Nehari manifold 


In this section, we study the Nehari manifold corresponding to the following equation: 


-Am - —- = Kaj^\u\’^ inR^, 

'" = Kf3j^\u\‘^\v\>^-'^v inR^, 
iu,v) G X 


(4.1) 
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For (u, v) € we define the norm 


{u,v)\y = {\\ur+\\vr) 


2 i 2 


where ||tt|| := ( \S/u\'^dx) ^ for u G Dq’^(R:!^). A pair of function {u,v) is said 

to be a weak solution of ( 14.11 ) if and only if 


f IliP (^l) 

VuVifi + VvVif2dx — A / -^^- dx — 




dx 


JR^ 

Jr^ 


/R^ 


|u|“|v|^ ^Vip2 


IR'I 


dx = 0 for all {ipi,ip 2 ) € 


The corresponding energy functional of problem (14.1b is defined 


as 


= ^a(u,u) - ~ 


for all {u, v) € where 


'a{u,v) := \\{u,v)\\%, 

< b{u, v) := A 4^ ^^^dx + p ^^dx, 


S.+ N |x|'’2 

We consider the corresponding Nehari manifold 

J\f := {(u, v) G ^\(0,0)1 J(it, v) = 0} 

where 


(4.2) 


J{u,v) :=($'(«, u), {u,v)) 

=a{u, v) — b{u, v) — K{a + P)c{u, v) 

and $'(u, v) denotes the Frechet derivative of 4> at {u, v) and (•, •) is the duality product 
between ^ and its dual space 

Lemma 4.1. Assume si,S 2 G (0,2),A,/r G (0,+oo),a > l,/3 > 1 and a + P = 
2 *(s 2 ). Then for any (u,v) G ^\{(0,0)}, there exists a unique t = > 0 such 

that t{u, v) = (tu, tv) G Af if one of the following assumptions is satisfied: 

(i) K > 0. 

(ii) K < 0 and S 2 > Si. 

(Hi) S 2 = Si and k < 0 with |k| small enough. 

Moreover, Af is closed and bounded away from 0. 
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Proof. For any {u,v) S we use the notations defined by (14.21) and we will write 
them as a, b, c for simplicity. It is easy to see that for any (u, v) f (0,0), we have 
a > 0, 6 > 0, c > 0. For any (u, v) £ ^\{(0,0)} and f > 0, we have 


^(tu,tv) =-at^ -T— -bt^ — Kct^ 

^ 2 2 *(si) 

Denote := where 


(4.3) 


g{f) = 6 f 2 *(-i )-2 - a. 


For the cases of (i) and {ii), it is easy to see that p(+oo) = +oo and g{0) = —a < 0. 
Also for the case (in), by the Young inequality, one can prove that there exists some 
C > 0 such that c{u, v) < Cb{u, v) for all {u, v) £ Thus, for the case of S 2 = si, 
if K < 0 with I K I small enough, we obtain that 

b{u, v) + 2*{s2)kc{u, u) > 0 for all (u, v) £ ^\{(0,0)}. (4.4) 


Hence, we also have p(+oo) = +c» and p(0) = —a < 0. 

Thus, we obtain that there exists some f > 0 such that g{t) = 0 due to the continuity 
of g{t). It follows that tu £ Af. By the Hardy-Sobolev inequality and the Young 
inequality, there exists some C > 0 such that 

b{u,v) < C\\{u,v)\\%'^''^\ c{u,v) < C\\{u,v)\\%^"‘^\ 

Let (m, v) £ Af, since 2*(si) > 2, i = 1,2, we have 

( 2*(si ) 2 *(s2) \ 

which implies that there exists some Jo > 0 such that 

||(m, u)||® = 02 > Jo for all (u, v) £ M. (4.5) 


Thus, Af is bounded away from (0, 0) and obviously, Af is closed. 

For any (u, v) f (0,0), let := inf{f|p(f) = 0, f > 0}. Then we see that fo > 0 
and g{ttf) = 0. Without loss of generality, we may assume that fo = 1, that is, g{t) < 0 
for 0 < f < 1 and p(l) = 0 = J + k2*(s2)c — a. We note that 

g'{t) =(2*(si) - 2)6f2*Li)-3 k 2 *(s 2 )( 2 *(s 2 ) - 2)cf2*(«)-3_ 


(i) If K > 0, then g'{f) > 0 for all t > 0. 

(a) If K < 0, S 2 > si, recalling that 0 = J + k 2 *(s 2 )c — a, we have 

g'(t) =(2*(si) - 2 )M 2 *Li )-3 k; 2 *(s 2 )( 2 *(s 2 ) - 2 )cf 2 *(« 2)-3 

= ( 2 *(si) - 2 )Jf 2 *(si)- 2 *L 2 ) k 2 *(s 2 )( 2 *(s 2 ) - 2)c 

= : fi(f)f2*(-2)-3^ 


(s2)-3 
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where 


h{t) := ( 2 *(si) - 2 )&t 2 *(«i)- 2 *(- 2 ) ^ k 2 *{s 2 ){ 2 *{s 2 ) - 2 )c. 
When t > 1, we have 

h{t) >{2*{si) - 2)6 + k 2 *{s 2 ){ 2 *{s 2 ) - 2)c 

= ( 2 *(si) - 2 ) (a - k 2 *{s 2 )c) + k 2 *{s 2 ){ 2 *{s 2 ) - 2)c 
= {2*{si) - 2)a - k 2 *(s 2 )( 2 *{si) - 2 *{s 2 ))c 
> 0 . 


Hence, g'{t) > 0 for alH > 1. 

{in) If K < 0, S 2 = si, similar to the arguments as case (ii) above, we know 
h{t) = (2*(si) - 2) (6 + 2*(si)kc) > 0 
when K is small enough by ( 14.4b . 


The arguments above imply that g{t) > 0 for f > 1. Hence, f = 1 is the unique 
solution of g{t) = 0. It follows that for any {u,v) ^ (0,0), there exists a unique 
t{u,v) > 0 such that (u, v) G Af and 

f(„.„)u) = max tv). 

□ 


Lemma 4.2. Under the assumptions of Lemma 14.71 any {PS)m sequence of ^{u, v) 
i.e., 

j 4>(u„,u„) -)■ m 

-)■ 0 in 

is bounded in 

Proof. Let {(u„,u„)} C be a {PS)m sequence of <!)(«,u). We tend to use the 
previous marks a, b, c and denote a(M„, Vn), b{un, u„), c('u„, ?;„) by a„, bn, Cn for the 
simplicity. Then we have 


Vn) = ia„ - - KCn=m + o(l) 


and 


J {tin, tin) — Hn bn k{o^ [3)Cn — t?(l) || {Un, Vn) || 
(1) If K > 0, for the case of S 2 < si, we have 

m + 0 ( 1)(1 + \\{Un,Vn)\\&) 

1 


(4.6) 

(4.7) 


— ^{lln,ttn) p.,, / ,J{tln,ttn) 
2 *{si) 


^ A _ 

4 2 *{s,y' 

>(i- 


. 2 *(s 2 ) 

^ 2 *(si) 


-l)t 


1 


^2 2 *{siy 


[liji , Vn j II ^ 
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and for the case of S2 > si, we have 


m + 0(l)(l + \\iUn,Vn)\\&) 


= ^{Un,Vn) - 


2 *{s2 ) 


J{,^n 7 t^n) 


A 


'2 2*(s2) 

- ^2 " 2 *(s2 )' 


1 ^ t 1 

jClri + ( 


2*(s 2) 2*(si) 


)b. 


[Un: '^n)\l 


(2) If Ai: < 0, S2 > Si, similarly we obtain that 

m + 0{1){1 + \\{Un,Vn)\\3>) > WiUn, Vn)\\%■ 


Based on the above arguments, we can see that { (m„, u„)} is bounded in □ 

Lemma 4.3. Under the assumptions of Lemma 14.71 let {{un, t'n)} Cl M be a {PS)c 
sequence /or i.e., 4)(m„, u„) —?► c and w„) —?► 0 in Then {{un, Vn)} 

is also a {PS)c sequence for $. 

Proof. For any (m, v) G J\f, we will follow the previous marks a, b, c defined by ( I4.21 l. 
Then we have 

a — b — k 2 *(s 2 )c = 0 

and 

{J'(u, v), {u, v)) =2a — 2*{si)b — k(2*(s2))^c. 

(l)If K > 0, 


(J'(u, v), (u, v)) =2[b + k2*(s2)c] — 2*(si)6 — fc(2*(s2))^c 
= [2-2*(si)]6+ ( 2 - 2 *(s 2 )) 2 *(s 2 )kc 
< max{2 — 2*(si), 2 — 2 *(s 2 )} + 2 *(s 2 )kc] 

= max{2 - 2*(si), 2 - 2 *(s 2 )}a. 


(2) If K < 0, S2 > Si, 

u), (u, v)) =2a — 2*(si) [a — /t(a + /3)c] — + PYc 

= [2 - 2*(si)]a + «;[2*(si) - a - /?] (a + /3)c 
<[2-2*(si)]a. 

Hence, by (I4.51 l. we obtain that 

{J' {u, u), (u, v)) < max{2 — 2*(si), 2 — a — I3}6 q < 0 for all (it, v) G Af, (4.8) 

where Jq is given by (14.5b . By the similar arguments as in Lemma l4!2l we can prove 
that {(it„, u„)} is bounded in Let {tn} C K. be a sequence of multipliers satisfying 

4^ — 4* Ijt/{^U n7 Vji^ -f tnJ (^'Uji^Vnf 
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Testing by (u„, w„), we obtain that 

i'^n 5 t’n) 7 (^n ; t^n)) ^ 0- 

Recalling ( 14.81 ). we obtain 0. We can also have that J'{un, Vn) is bounded due to 
the boundedness of (u„, u„). Hence, it follows that w„) —>■ 0 in □ 


Define 

Co := inf $(u,u) (4.9) 


and 



1 




where s^ax ■= max{si,S2}- From the arguments in the proof of Lemma \4^ we 
obtain that 

co>vUu,v)\\%. (4.10) 

Combined with Lemma l4n we have 


Co > rjSl, 


(4.11) 


where bo is given by (14.5b . If uiq is achieved by some (it, v) G Af, then (it, v) is a 
ground state solution of (13.3b . 


5 Nonexistence of nontrivial ground state solution 

In this section, we continue to study the equation (14.1b . 

Definition 5.1. In the sequel, we call (it,u) nontrivial ijf u ^ 0 and u ^ 0, and call 
(it, v) semi-trivial iff either u = 0 or v = 0 but not all zero. 

We obtain the nonexistence of nontrivial ground state solution of (Hall. i.e., the least 
energy co := inf(„ '!’('«, v) defined in ( 14.9b can only be attained by semi-trivial 
pairs. Denote 


:= inf I - 


Irn IVupdx 


(/t 






dx) 


uGDt 


2*(s) 


(R?)\{0}}. 


(5.1) 


By the result of Egnell ifTol . ps,^(K(^) is achieved and the extremals are parallel to 
U(x), a ground state solution of the following problem: 



(5.2) 
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Define the functional 


A 

2 *(si) 


(5.3) 


= ^ [ iVupda; - 




dx. 


Then a direct computation shows that u is a least energy critical point of 'I'a if and only 

u = Ux-= [/, (5.4) 

A 

where [/ is a ground state solution of (I5.21 i. And the corresponding ground state value 
is denoted by 


mx = '^x{Ux) 


2 *(si) 


]{dsA^+)) 


2*(«1 ) 

•{ = l )-2 




Then we see that rax is decreasing by A and 


Co < min{mA,m^}, 


(5.5) 


(5.6) 


where cq is defined by ( 14.9b . 

Theorem 5.1. Assume that a + f3 = 2* ( 32 )- If one of the following conditions is 
satisfied: 

(i) K < 0 and S 2 > Si; 

{ii) minjo;, fi} > 2, S 2 > Si and k > 0 small enough, 

then we have 

Co = min{mA,m^}. 

Moreover, cq is achieved by and only by semitrivial solution 


r(17A,0) 

<(0,t/^) i/'A<p, 

[(17A,0)or(0,t7A) = 

where Ux,Uf_t are defined by (EH). 

Remark 5.1. Theorem 15.71 means that the system SJ} has only semi-trivial ground 
state under the hypotheses of the theorem. 

Remark 5.2. If Si = S 2 = s G (0,2), min{a,/3} > 2, we must have N = 3. In this 
case, the assumption that “k is small enough” can be removed (see Theorem \7.4\l . 

Proof Without loss of generality, we only prove the case of A > /r. By (15.6b . we see 
that Co < TO A. By (14.1 lb . we also have cq > 0. Now, we proceed by contradiction. 
Assume that cq is achieved by some (u, v) G & such that u ^ 0, u 0. Without loss 
of generality, we may assume that m > 0 , u > 0 since co is the least energy. 
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(i) If K < 0, we obtain that 


/R" 


|Vwp-A 


| 2 *(si) 


-dx = na 


\u\ \v 


1 ^ 


dx < 0 . 


/r" FI' 


/r« FI' 


Recalling that 


if M 7 ^ 0 , we obtain that 




-dx 


^ < llwll^ 


R^ FI 


/r" F 




A 


Then by S 2 > si, 

1 


'2 2 *(s 2 ) 

>fi- ^ 


)\\ur+i 


2 *{s 2 ) 2*(si)^ 7 rn |x|«i 


|2*(si) 


-dx 




^2 2 *(s2 ) 

1 1 


N 

Si 


|2*(si) 


^2*(s 2) 2*(si)^ Jb^n |a;|®i 


Fr^ 

|,,| 2 *(si) 


-dx) + 


-dx 


>d- 


2 2 *(s 2 ) 

1 


1 + 


2*(s 2) 2*(si) 


A” (Msi (R+ 


= (^-^)A--^(Msi«)^ 


= m\. 

Similarly, if v 7 ^ 0, we have 


.| 2 *(si) 




R^ FI' 


and 


,1 1 


^2 2*(s2)' 


'2*(s 2) 2*(si)^ Jbn |x|*i 


.| 2 *(si) 


(5.7) 


(5.8) 


-dx > rria > m\. 
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Then, 


Cn =<&(«, v) = (— — - r)a(u, v) + ( r — — ^ v) 

^ ^ ^2 2 *{ s 2 )’ ^ ''2*(s 2) 2*(si)^ ^ ^ 


_ I _) 

'2 2 *(s2 )^ 

A 1 


1 


'2*(s2) 2*is,)’ U. 1x^1 


|2*(si) 


-dx 


A 2 *(s 2)' 


''2 *(s 2) 2*(si)^ Jt^n |a;|*i 


,|2*(si) 


-dx 


> 


{ m\ if = 0 

mx + ifv^O. 


Hence, cq = mx is proved and we see that r; = 0, i.e., (m, v) = (C/a, 0). 
{ii) If K > 0, we denote 


|2*(si) 


a := 


Then we have 


Cr" FI 


a < 


-dx, S := 


,|2*(si) 


-dx. 


/r;;^ fI 




A 


If not, apply the above similar arguments, we have 


1 


2*(si) 


-dx > mx, 


a contradiction. Similarly, we also have 

Similar to the arguments of Lemma 14. II we have <I>(u,?;) > (| — ■fr^)\\{u,v)\\'^. 
Hence, ||M||^,||f||^ < {\ — 2*(s2) By Corollary 12.21 under the assumption of 

min{a, /3} [j^Ig^|(2-si) ^ choose some proper rji > 2 ,t ]2 > 2 and C > 0 


such that 


/r" FI 


-dx < = Ccr^W 


and 


,|/3 


Cr;^ FI 




It follows that there exists some C > 0 such that 


and that 


Psi(R+— Act < kCct^fsi) 


^^sA^+)6^^ -fi5< kCS^^. 


(5.9) 

(5.10) 

(5.11) 

(5.12) 
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Define gi : R_|_ R.+ , z =1,2 with 


and 


2* (Sj ) —2 7)1 -2 

gi{t) := \t + kCI 


2*(ai )-2 7 , 2-2 

g2{t) := g.t 


Since gi,g 2 > 2, gi(i) is strictly increasing in terms of f. It is easy to check that there 
exists some kq > 0 such that when k < kq, a direct calculation shows that 


9i 




<t‘,AK) 


and 


92 




I I ^ 2 ('^l) 

2 »(si )-2 


< 9 




Hence, if cr 7 ^ 0,5 7 ^ 0, by (15.1 lb and (15.12b . we obtain that 


1 

a>-( 


2 *(«i) 

±1.1 2»(si)-2 


and 




Then 


1 


2 *(s 2 ) 

1 




1 


1 


>(i- 

-4 2*{s2) 
(1 1 
■^2 " 2 *(s 2 ) 


N 


2*{s2) 2*{s,) 

1 


|2*(si) 


-dx 


R" Ft 


^)^ 2 W + ( 


1 


2 *(s 2 ) 2 *(si) 


) Act 






±Zl 2 *(si)- 2 1 2-(si) 


'^^2*(s 2) 2*(si)^^2^ A 


1 


> 1 ( 1 - 


1 


2^2 2*(si) 


)A- 


2 *(<!i )-2 


2 *(ai) 

) 2 *(<.i )-2 


1 

= ^m,. 


Similarly, we have 

/I 1 


2 *(s 2 ) 


)\\ vr +{ 


1 1 ^ /■ , 1 1 

—T ~ 7r-7 — 7)9 / —:—:- dx > > -m\. 

2*{s2) 2*{si)’^J^n 2 2 ^ 
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Thus, 


^ ^ 4 2*{s2)' 


1 


1 


^2*(s 2) 2*(si) 






-dx 


/R" 


1 


2 *(s 2 ) 


)\\vr 


^2*(s 2) 2*(si) 




U|2*(si) 


/R" 




-dx 


>m\, 


a contradiction. 

The arguments above imply that cr = 0 or i5 = 0, i.e., u = 0 or u = 0. If u = 0, 
then u ^ 0 is a critical point of and then <!>(«, v) = > rn.^ > m\, also a 

contradiction. Thus, we obtain that u 7 ^ 0, u = 0. Hence m is a critical point of 
and Co = ^{u, v) = 'I'a(m) > m\. Then, we have cq = m\ and u = U\. □ 

Remark 5.3. We remark that the Theorem \5.1\ ofthis section is valid for any cone O. 


6 Preliminaries for the existence results 

Remark 6.1. Without loss of generality, we only consider the case ofVl = We 
remark that the results of this section are still valid for any domain H as long as gisi {^) 
is attained (e.g. H is a cone). 

Since the system (14. Il l possesses semitrivial solution (m, v), we are interested in 
the nontrivial solutions. Firstly, we recall the following result due to Ghoussoub and 
Robert |[T3] Theorem 1.2] (see also ([TSl Lemma 2.1], ([191 Lemma 2.6]) for the scalar 
equation. 

Lemma 6.1. (' 1173] Theorem 1.2]) Let u € be an entire solution to the 

problem 

+ ( 6 . 1 ) 

IM > 0 in and u = 0 on . 

Then, the following hold: 

(i) _ 

rueC' 2 (K^ ifs,<l + j^, 

< u G for all 0<f3<lifsi = l-\--^, 

forall0< P < ^^^^ifsi>l + j^. 

(ii) There is a constant C such that |u(a;)| < (7(1 + and |Vu{a;)| < (7(1 + 

(Hi) u{x',xn) is axially symmetric with respect to the x^-axis, i.e., u{x',xn) = 
m(|x'|, cctv), where x' = {xi, • • • , xn-i). 
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_ , 2*(si)-2 

6.1 Existence of positive solution for the case : A = //(^) 2 

The following result is essentially due to iflTl Theorem 1.2]: 

Lemma 6.2. Let N > 3, si,S 2 G (0, 2), A > 0, a > 1, /3 > 1 and a + /? = 2*(s2)- 
Then the following problem 


—Aw — A 
w{x) e D, 




2*(a2)-l 


g’^(R:!^), w{x) > 0 in 


= 0 in 


( 6 . 2 ) 


has a least-energy solution provided further one of the following holds: 

(i) 0 < Si < S 2 < 2 and k G R. 

(a) Si > S 2 and k > 0 . 

(iii) Si = S 2 and k > ~Ai(^)^. 

Remark 6.2. The case of k, = 0 or si = S 2 with k > ~Ai(^)^, (I6.21 i is essentially 
the problem (16.1b . And the existence result was firstly given by Egnell 4701? . 

Corollary 6.1. Let N > 3, si,S 2 G (0,2),/r > 0, k 0, a > l,/3 > 1 and 
a + /3 = 2 *(s 2 ). If X = then {w, is a positive solution of ( 13.3b 

provided further one of the following holds: 

{i) 0 < Si < S 2 < 2 and k G M\{ 0 }. 

(ii) Si > S 2 and k > 0 . 

(iii) Si = S 2 and k > 

Here w is a least-energy solution of dO) . 

Proof This proof can be got through via a direct computation. We omit the details. □ 

Corollary 6.2. Assume that N > 3,a > l,/3 > 1 ,q; + /3 = 2 *(s 2 ), A = p-(^) ^ > 
0 and one of the following holds 

(i) 0 < Si < S2 < 2 , K < 0 , 

(ii) Si = S2 G (0, 2), -Ai(|)^ < K < 0, 

(iii) minja, fi} > 2, S 2 > Si and k > 0 small enough. 

Then (w, \J~^w) is a positive solution to equation (13.3b but problem (13.3b has no non¬ 
trivial ground state solution. 

Proof. It is a straightforward consequence of Theorem IS.ll and Corollary 16. II □ 
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6.2 Estimation on the upper bound of cq := inf $(«, v) 

(u,v)€Af 

In order to prove the existence of positive ground state solution to the equation (14. li . 
we have to give an estimation on the upper bound of cq, including the cases of si = S2 
and Si ^ S 2 - 

Let 1 < Of, 1 < /3, a + /3 = 2*{ 32 )- Let u := U\ be a function defined by ( 15.4i . 
Then we have u > 0 in and 

c{u,v):= j dx > 0, y V e 

Assume that the assumptions of Lemma 144] are satisfied, i.e., one of the following 
holds: 


(i) K > 0, 

(ii) K < 0 and S 2 > si, 


(iii) S 2 = Si and k < 0 small enough, 

then we see that for any e G M, there exists a unique positive number t(e) > 0 such 
that (t{e)u, t{£)ev) G N. The function tie) : M 1 —>■ M+ is implicitly defined by the 
equation 


|uf = Ah 






2 *(si) 




2 *(si )-2 


■ K 2 *(s 2 )c(u,u)[f(e)] 


2*(s 2)-2. .p 


(6.3) 


We notice that f(0) = 1. Moreover, from the Implicit Function Theorem, it follows 
that tie) G C'^(R) and f'(e) = where 


Q.(e) :=[2*(si)-2] 

+ k 2 *(s 2 )[ 2 *(s 2 ) - 2]c(M,u)[f(e)] 


|2*(si) I 
2*(si).si I 


| 2 *(si) 




2*('<i)-3 


2 *(s 2 )- 3 | 1^ 


and 

P„(e) :=2||u|pe-2*(si)p|u|2.[^j]_^jf(e)]^ 

- K2*is2)f3ciu,v)[tie)]^ ‘^\e\^~'^e. 

Lemma 6.3. (The case of /3 < 2) Assume that l<a, l</3<2, a + /3 = 2*(s2) 
and one of the following holds: 

(i) K> 0 . 

(ii) K < 0 and S 2 > Si. 

(iii) S 2 = Si and k < 0 with |k| small enough. 
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Let U\ := 2 *(ai) where U is a ground state solution of (15.2b . Then 

(a) if K < 0, (u, 0) is a local minimum point o/$ in J\f. 


and 

Hence, 

where 


(b) if K > 0, then 

Co := inf ip) < ^{Ux,0) =\{Ux) = uix- 

(0,v)eAA 

Proof Let u := Ux and take any v G I?q’^(]R:!^)\{ 0}. When /3 < 2, we have 
Pvis) = —k2*(s2)/3c(u, n)|e|^“^£(l + o(l)) as e —>• 0 

g.(e) = ([2*(si)-2]A|«|^:j:;J^^J(l + o(l)) ase^O. 

t'{e) = -M{v)l3\ef~'^e[l + o(l)), 

M(.) :=- 

By the Taylor formula, we obtain that 

t{£) = 1 - M{v)\ef{l + o(l)), 

= l-2*{s,)M{v)\sf {1 + 0 ( 1 )), 

and 

[t(s)f^^^^ = l-2*(s2)Miv)\ef (1 + 0 ( 1 )). 

Noting that for any (f, p) € Af, we have 

P) = {\ - P) + - -Kc((j), p), 

where &(</>, p)^ c(4>, p) are defined by (14.2b . Thus, 

4>(f(e)M, f(e)eu) — $(m, 0) 

^ ^ |2*(*i) Lt=-M2*(«i) _L ,,l„|2*(si) 








— ‘^Kc(u,v)[t(e)Y 


=(^ - + «(i)) 

_^ 2 _(s^—^Kc(u,t;)[f(e)]^ 

= - ^^^-^c(u,u)|£|^(l + o(l)) + —^Kc(u,u)(l + o(l))|e|'^ 

= -K|e|^c(M,u)(l + o(l)), 

which implies the results since c(u, u) > 0 for any 0 v € Z3q’^ 


□ 
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Lemma 6.4. ("the case of /3 > 2) Assume that 1 < a,2 < f3,a + (3 = 2 *(s 2 ) and one 
of the following holds: 

(i) K > 0. 

(ii) K < 0 and S 2 > si. 

(Hi) S 2 = Si and k < 0 with |k| small enough. 

Let U\ := where U is a ground state solution of (15.2b . Then 

{U\, 0) is a local minimum point 0 /$ in Af. 

Proof Let u := U\ and take any v G Dg’^(]R:!^)\{0}. When ^ > 2, we have 


P„(e) = 2||i;f£(l + 0 ( 1 )) 

and 

Q4e) = ([ 2 *(si) - 2 ]A|^x|^:[:;J_,J (l + o(l)). 

Hence, 

t\e) = 2M{v)e(l + o(l)), 

where 


M{v) 


Ikll 


2 


[2*(si)-2]A|m| 


2 *(si) 

2*(si),si 


By the Taylor formula, we obtain that 


f(e) = 1 + M(z;)|£p(l + 0 ( 1 )), 

[t(e)]^*(^^) = l + 2*(si)M(u)|£p(l + o(l)), 

and 

[t{e)f^^^'> = l + 2*{s2)MivM{l + oil)). 

Hence a direct computation shows that 

^(t(^£)u,t(^£)ev) — <I)(m, 0) 

= (^ - [2*(^l)^(-)] kl^(l + 0(1)) + o(|£n 

=^lkfkP(i + o(i)) 

>0 when e is small enough. 


Dehne rji := inf ||u||^, where 

v^S 




| a ;|®2 


dx = 1}. 


□ 
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We note that by the Hardy-Sobolev inequality, U\ G then by the 

Holder inequality, dx is well defined for all v G Dq’‘^{M.^) when a = 

2 *{s2 )-2. 

Define (^, ij}) := then it is easy to check that (•, •) is an inner prod¬ 

uct. We say that cj) and ij) are orthogonal if and only if (</), ij}) = 0. Then we have the 
following result; 

Lemma 6.5. Assume that 1 < a = 2*(s2) — 2,/3 = 2, then there exists pi > 0 and 
some 0 < u G Dq’‘^(R.^) such that 


-Au = 771 ^^ 1 ^ 5 - 1 ; ifi 

7 ; = 0 on 


(6.4) 


Furthermore, the eigenvalue 771 is simple and satisfying 


/ ' —p-f-LLdx < — \\v\\‘^ for all v G Dq’^(IR^). 


(6.5) 


In particular, if si = S 2 = s, we have 


m = A, 


( 6 . 6 ) 


and it is only attained by v = U\. 

Proof It is easy to see that 771 > 0 under our assumptions. Let {vn} C S be such 
that llu^lp — 771 . Then {u^} is bounded in Going to a subsequence if 

necessary, we may assume that Vn vo in and Vn vo a.e. in By 

Holder inequality, we have 


L 


,a„,2 


a ,,2 


U V" — U V, 


-dx 


< 


,a-V2 


-dx 


A FI' 


Ivl - 


0 


as |A| —0 due to the absolute continuity of the integral and the boundness of Vn- 
Similarly, we also have that is a tight sequence, i.e., for £ > 0, there exists 

some i?e > 0 such that 


-dx 


/RynB|(^ FI 


< £ uniformly for all n G N. 


(6.7) 


Combine with the Egoroff Theorem, it is easy to prove that 


[ ^dx^! 




U V, 


R" Fr^ 


-dx. 


( 6 . 8 ) 


Hence, we prove that 


D. 


1,2 fnN 


) I—>■ R with x('t') = 


'r« FE" 


-dx 


32 

















is weak continuous, which implies that ^ is weak closed. Hence, uq € S and we have 


||uo|P < liminf ||u„|p = pi. 

n—¥oo 

On the other hand, by the definition of and vq € S, we have 


> m- 

Thus, Vo is a minimizer of ||t;|p constraint on S. It is easy to see that |tio| is also a 
minimizer. Hence, we may assume that uq > 0 without loss of generality. We see that 
there exists some Lagrange multiplier 77 e K such that 


-Avo = 77 




X 


S2 


It follows that 77 = 771. Since vq G we get that uq 7^ 0 and 771 > 0 . 


. Then the Brezis-Kato 


I I — 

Let a{x) := rji it is easy to see that a{x) £ 

theorem in ID implies that v € for all 1 < r < 00. Then vq € , 

for all 1 < r < 00. By the elliptic regularity theory, vq € Finally, by the 

maximum principle, we obtain that vq is positive. Finally, ( 16.511 is an easy conclusion 
from the definition of 771. 

It is standard to prove that 771 is simple, we omit the details. Next, we will compute 
the value of 771 when si = S2 = s. A direct computation shows that 


-AUx = \ 


U 


2*(s)-l 


(6.9) 


Testing ( 16.41 ) by U\, we have 

/ (Vu • WU\)dx = 771 
Testing ( 16.9b by v, we also have 

(VC/a • Vu)dx = A 


jja 

-r-^vU\dx. 


'r" FI' 


jjot 

Y^Uwdx. 


/R" 


/R" Ft 


Hence, 


im - A) 


u? 


Ft 


■U\vdx = 0 . 


Since v and U\ are positive, we obtain that 771 = A. 


( 6 . 10 ) 


( 6 . 11 ) 


( 6 . 12 ) 


□ 


Lemma 6.6. (the case of /3 = 2 ) Assume that 1 <a = 2 *(s2) — 2,/3 = 2 and one of 
the following holds: 

(i) K> 0. 

(ii) K < 0 and S2 > si. 
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(Hi) S 2 = Si and k < 0 with |k| small enough. 

Let U\ := 2 *(ni)-^ u, where U is a ground state solution of (15.2b . Then 

there exists a positive Lq = 2 »Ji 2 ) 

(a) if K < 0, (U\, 0) is a local minimum point o/<J> in Af. 


(b) ifO<K< ItQ, then (U\,0) is a local minimum point o/$ in Af. 


(c) if K > ItQ, then 


Co := inf ip) < ^{Ux,0) =x(Ux) = mx, 


where rjx := inf with 

uGH 


r ^ nl■2mw^ f 

:={.£D„ (1^): 


Proof. Let u := Ux and take any v S Df (R:!^)\{0}. In this case, we have 
Py[e) = 2(^||?;|p - K2*(s2)c(u,?;)^e(l + o(l)) 

Q„(e) = ([2*(si)-2]A|«|^:[::j,J(l + o(l)). 
t'{e)=2M{v)e{l + o{\)), 


and 

Hence, 

where 


M{v) := 


II^IP - k2*{s2)c{u,v) 

|2*(si 


Similar to the arguments above, we obtain that 

$(f(s)'u,f(s)su) - $(14,0 ) = - k 2 *(s 2 )c(ii,i;))|s |^(1 + o(l)). 

If K < 0, we obtain the result of (a). 

Since it > 0 is given, by Lemma|63] we have 


c(u,v) = / 

JR 


, |—<—||i;|p for all u G L>o’^(R(^). (6.13) 


Define 


kn := 


m 


2*(s2)' 
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Hence, when k < ko, we have 

||t!||^ — 2 *{s 2 )kc{u, t;) > 0 for all v € Z?q’^(M^)\{0}, 

which implies (b). For k >ko = ^, by the definition of rji, there exists some 
V € Dq’^(IR:?^)\{ 0} such that 

||t;||^ - k2*(s2)c(u,v) < 0. 

and then it follows (c). □ 

Summarize the above conclusions, we obtain the following result: 

Corollary 6.3. Assume that 1 <a,l </?,« + /?< 2*(s2) one of the following 
holds: 

(i) K > 0. 

(ii) K < 0 and S 2 > si- 

(Hi) S 2 = Si and k < Q with |k| small enough. 

Let U\ := U, where U is a ground state solution of (15.2b . 

(1) Assume that either K<0or/3>2or/3 = 2 but with k < 2 ^r^^’ 
is a local minimum point o/<i> in Af. 

(2) Assume that either j3 < 2 and k > 0 or /3 = 2 but with k > 2 *(l 2 ) ’ 

Co := inf $(</>, (^) < $(t/A, 0) = ^'a(C/a) = m-A, 

(0,v)gaa 

where rji is defined as that in Lemma 1631 In particular, rji = X ifsi = S 2 = s. 

Apply the similar arguments, we can obtain the following result: 

Corollary 6.4. Assume that 1 <«, 1 < fi,a. + fi < 2*{s2) and one of the following 
holds: 

(i) K > 0. 

(ii) K < 0 and S 2 > si. 

(Hi) S 2 = Si and k < 0 with |k| small enough. 

Let Ufj. := 2^1377^ jj^ where U is a ground state solution of ( 15.2b . Then 

(1) if either K<0 ora>2ora = 2 but with k < 2 ^fs 2 ) ’ local 

minimum point of $ in Af; 
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then 


(2) if a < 2 and k > 0 or a = 2 but with k > 2 ^{s 2 ) ’ 

Co = inf $((/>, v?) < $(0, = 


(0,v)GAA 




u £ D, 


where 772 := inf \\u\\‘^, and 0 := -i 
u£Q I 

particular rj 2 = whenever si = S 2 = s. 


1,2/m.Af 


) : / 




dx = In 


\ 


1 The case of si = §2 = s G (0, 2): nontrivial ground 
state and uniqueness; sharp constant ex¬ 

istence of infinitely many sign-changing solutions 

In this section, we focus on the case of si = S 2 := s € (0,2); the case si S 2 will be 
studied in the forthcoming paper (Part II). That is, we study the following problem 




—Art — A 
—Av — /i 


\x\‘ 

|„|2*W-2, 




|xp ^|x| 

{u,v) e ^ := X £>o’^(12), 


in n, 
in 12, 


(7.1) 


where 12 is a cone in R^(especially, 12 = or 12 = R(^) or 12 C is an open do¬ 
main but 0 ^ 12. In this section, we are aim to study the existence of nontrivial ground 
state solution to the system ItTI Thanks to the fact of that si = S 2 , we shall obtain 
further results: the uniqueness of the nontrivial ground state solution; the relationship 
between the sharp constant and the domain 12; the existence of infinitely 

many sign-changing solutions to svstem lTTTI Finally, we will explore some approaches 
for studying the sharp constant when 12 is not necessarily a cone. 

Noting that for the special case si = S 2 = s £ (0,2) and a + P = 2*(s), the 
nonlinearities are homogeneous which enable us to define the following constant 

Sc.,p,xA^)-= inf_-AlA-^^- U - 


2^ 


where 

^ := {(u, v) £ : 


u 


Gl' 


2*(s) 




+ M" 


|a;p |a;|- 

The above constant determines the following kind of inequalities: 


(7.2) 


-f 2* dx>0}. (7.3) 




,|2*(-) \v 


2*(s) 


+ 2*(s)^ 




)dx'j 


2^ 


< [ {\\7u\^+ \Vv\^)dx 

Jn 


(7.4) 
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for (m, v) € ^ or which can be viewed as the double-variable CKN inequality. To 
the best of our knowledge, such kind of inequality and its sharp constant with extremal 
functions have not been studied before. 


Denote 


,(n) := inf I 


iVitPdx 


: u G D, 




(7.5) 


then /rs(0) can be attained when fl is a cone in and 0 < s < 2 (see nni). Noting 

that £)Q’^(f2) X {0} C and {0} x Dq’^(O) C by the definition, we have that 

-S'a./ 3 ,A,/i(f^) < (max{A,/r}) (7.6) 

By Young’s inequality, it is easy to see that > 0. The following statement 

is obvious. 

Proposition 7.1. Assume that (u, v) is an extremal of such that 

f +2*(s)/t|ur|t;|'5]dx= 1 

JQ Fl 


and 


Then 


(|Vu|2+|Vup)dx = 5„.;3.A.,.(0). 


is a ground state solution to problem (TTTI i and the corresponding energy 

Co = V') = (P [^o,/3.A.4f^)] • 

Lemma 7.1. Assume that a > 0, /3 > 0, A > 0, /i > 0, then there exists a best constant 

k(q;,^, A,/r) = (a+ /3)(-)^(^)^ (7.7) 

a p 


such that 

K{a,(3,\,p) f \u\°‘\v\^di' < X ( dvp f \v\°'^^di' 

JO J Q, J Vt 

for all {u,v) G d;/) x dz/). 

Proof By Young’s inequality with e, 

xy < £x^ + C{£)y'^ {x,y>Q,£>Q) (7.8) 
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where ^ ^ = 1,C{£) = (ep) Take p = and 

X = \u\°‘,y = \v\^,£ = —— , 

a + p p 

then we obtain that 

\unvf < , I , , (A|7rr+/^ + Plvl^^+P). (7.9) 

Hence, for all {u,v) G L°‘~^^{n,di') x {ft, diy), we have 

K{a,P,X,p) ( \u\°‘\v\^dv < \ f \u\°‘~^^di'+ p ( (7.10) 

Jvl Jq. Jvl 


And we note that when (m, tu) with t = {-^) “+'’, the constant «(«, /3, A, p) is at- 

pa 

tained. Hence, K{a, /?, A, p) is the best constant. □ 


Lemma 7.2. Assume k < 0. Then 

Sa,i3,x,i,i^) = (max{A,/r}) 

In particular, in this case *S'a,^,A,/x(^) can only be attained by the following semi-trivial 
pairs: 

({U,0) ifX>p; 

^(0,C/) ifX<p; 

[{U,0)or{0,U) ifX = p; 

where U is an extremal function of ps(fd). Hence, the ground state to problem (17.1b 
can only be attained by semi-trivial pairs. 

Proof. By (17.6b . we only need to prove the reverse inequality. Indeed, for any (u, v) G 
we have 


In + |Vu|2)fia: 


> 


In + \'^v\^)dx 

fr 

lap +1^ la;!* j 


2^ 


> (max{A, p}) 


2^ . 


In (|Vu|2 + |Vt;|2)dx 

2 

WTJJ 




>(max{X,p}) ^^ps(fl) 
>(max{X,p}) ps(fl). 


(/, “ 


2*(s) 


n |ai|® 






(7.11) 
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By taking the infimum over we obtain that 

> (max{A,/r}) 

Moreover, by the processes of ( 17.1 lb . we see that Sa,i 3 ,\,fj,{^) is only achieved by 


(U, 0) if A > Ai; 

(0,C/) if A < ax; 

^ {U, 0) or (0, U) if A = 


where U is an extremal function of Ars(Al). 

Proposition 7.2. 9 = ^\{{0,0)}whenK> 

a p 


□ 


.A, 


Proof. By Lemma lTn if k > —(—) =*'('») then there exists some C > 0 such 

a p 

that 




+2*(s)k 


\u\ \v 


1 ^ 


)dx > C(^ 


Thereby this proposition is proved. 




)dx'j. 

(7.12) 

□ 


Remark 7.1. By the Lemma [7 !^ and Pronosition \7.2\ we know that, when 0 > At > 
_(A)S’x ;!ihgfi fhg inequality ( 17.4b is meaningful but the sharp constant 
can be reached only by semi-trivial extrejnals. 


Note: In view of Lemma 17.21 Proposition 17.21 and Remark 17.11 we have to consider 
the case that k > 0. Therefore, throughout the remaining part of the current paper, we 
assume k > 0 . 

We obtain the following result: 

Theorem 7.1. Assume 12 is a cone in (especially, 12 = orLl = or 12 is an 
open domain with 0 ^ 12. IfO < s < 2, a > 1, /3 > 1, a + /3 = 2*(s) and k > 0, then 
is always achieved by some nonnegative function {u, v). 


Remark 7.2. Theorem U.h asserts that the constant can be attained by a 

nonnegative extremal function. But at this moment, we can not exclude the possibility 
of the semi-triviality of the extremal function. Fortunately, we will see further results 
in Theorem U.2\ below for the nontrivial extremal function. 


When we study the critical problem of a scalar equation, say (P), blow-up analysis 
is one of the classical method. Its ideas usually are as follows: Consider a modified 
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subcritical problem define on a convex domain, say {Pg), which approximates the prob¬ 
lem (P). Then study the existence of positive solution Ug to the modified problem (Pg) 
and the regularity of the positive solutions. By the standard Pohozaev identity, Ug must 
blow-up as e —0. Apply the standard blow-up arguments to obtain the existence of 
positive solution of (P). It follows that the set of solutions is nonempty. Therefore, 
one may take a minimizing sequence to approach the ground state. In this process, the 
approximation involves the domains and the format of the nonlinearities. However, for 
the system ITn the customary skills can not be applied directly. Since we want to get 
rid of the semi-trivial solution, the main obstacles lies in that we can not get the precise 
estimate about the limit of the least energy of the approximate problem. Therefore, in 
the current paper, we introduce a new approximation system where we just modify the 
singularity. 


7.1 Approximating the problems 


For any e > 0, set 



for |a;| < 1, 
for |x| > 1. 


(7.13) 


Lemma 7.3. Let e > 0. Then for any u € ag{x)\u\'^ is well defined 

and decreasing by e. 


Proof. Let ei > £2 > 0. By the definition of ag{x), it is easy to obtain the result by 
noting that ag^{x) < ag.^{x) < ao{x). □ 

We also note that for any compact set Hi C H such that 0 ^ Hi, ag{x) —?► ao(x) 
uniformly on Hi as e 0. For any fixed e > 0, we consider the ground state solution 
to the following problem: 


{ —Am — Xag{x)\u\‘^''^^'i~‘^u = Kaag{x)\u\°‘~‘^u\v\^ in H, 

—Av — fiag{x)\v\‘^''^'^'i~'^v = Kfiag{x)\u\°'\v\^~'^v in H, (7.14) 

K > 0, {u,v) G ^ := i7g’^(H) X i7g’^(H), 

Consider the following variational problem 


min f (iVwp + \\7v\'^)dx 

Jo, 

s.t. f agjx) [A|itp -I-/rjup -I-2*(s)K|'u|“|z;|^](ia: = 1. 

Jn 


p X minimize value of ( 17.151 1, then we have: 


We let SI 

Lemma 7.4. The constant ^ ^ ii(^) increasing with respect to e and 


£->■0 + 


(7.15) 

(7.16) 
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Proof. By the definition of 5® ^ ^{Vi) and a^ix), it is easy to see that 

^ '^^/ 3 .A,/.(^) for aly £1 > £2 > 0. 

Hence, we have 

liminfS'^ ^ ;^ (H) > (7.17) 

£-> 0 + 

On the other hand, for any rj > 0, there exists {u, v) G Cf^{VL) x Cf^{VL) such that 
f ao(x)[A|Mp + 2*{s)K\u\°‘\vf]dx = 1 


and 


[ (iVup + \Vv\‘^)dx < Sa,/3,\,f,i^) + V- 
JQ 


Since ae{x) —>■ ao(a;) in L°°{supp{u)) as £ —>■ 0, we obtain that 


limsupS'^_^;,_^(0) 

£->■ 0 + 


= lim sup 
£->■ 0 + 


In + |Vt;|2)da; 


2^ 


^ ae(x) [A|mP*(®) + + 2*{s)K\u\°^\v\P\dx^ 

In + |Vup)da: 


(In ^o(^) [A|mP*(s) + /i|?;|2*(s) + 2 *(s)k|m|“|u|^] dx'j 

= f (|Vup + |Vt;p)dx 

Jn 

By the arbitrariness of p, we have 




limsupS'® _^ (7.18) 

£—>■ 0 + 

Thus the proof is completed by (17.171) and (17.181) . □ 

Let LP{n, ag{x)dx) denote the space of LP-integrable functions with respect to the 
measure ae{x)dx and the corresponding norm is indicated by 

|u|p,e := (/ aeixfufdx)”, p>l. 

Jn 

Then we have the following compact embedding result; 

Lemma 7.5. For any e € (0, s), the embedding {ft) =-;> ^'^^{n,ae{x)dx) is 

compact. 
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Proof. Let {u„} C Z3g’^(f2) be a bounded sequence. Up to a subsequence, we may 
assume that ^ u in Z?g’^(U) and —?► u a.e. in VL. Then for any i? > 1, we have 



ae{x)\u\^ < 



ao|up 0, 


(7.19) 


uniformly for all n as i? —+oo. 

Noting that 2*(s) < 2*(s — e),2*(s) < 2* := by Rellich-Kondrachov 

compact theorem, we have 

lim [ as{x)\un — up = 0. (7.20) 

JnnBR 

By (17.19b and (17.20b . we prove this Lemma. □ 

Lemma 7.6. For any e € (0, s), p attained by some extremal {ug, Vg), 

i.e., 

[ a£(x)[A|uep**^®) + 2*{s)K\ug\°‘\vgf]dx = 1 

Jn 

and 

f (|Vu,p + |Vz;,p)dx = 

Jn 

Moreover, (ug, Vg) satisfies the following equation: 


{ —Au = S'® ^ ^(U) ^Aae(x)|up + Kaae(x)|M|“ in Ft, 

-Av = + K/3ae(x)|u|“|u|^“^u^ in U, 

u > 0,v > 0, (u,v) € ^ := Zlp’^(U) X Z7Q’^(r2), 

(7.21) 


Proof Let {u„ g} C ^ be a minimizing sequence, i.e., 

/ ae(x)[A|u„,ep + ft|wTi.£p + 2*(s)K|u„,e|“|z;„,ep]dx = 1 


and 


(|Vu„,ep + |VM„,ep)dx -(> S^ p x^^i^) as n +oo. 


Then we see that {un,g} and {vn,g} are bounded in i9g’^(r2). By Lemma l731 we see 
that up to a subsequence Un,g —Ug and Vn,e Vg in 1*1 (U, ag{x)dx). Hence 


t{x) [A|uep*(®) + 2*{s)K\ug\°'\vg\^]dx = 1. 


Then by the dehnition of p ^ we have 


/ (|Vu,p + |Vz;,p)dx > S^^^p^xA^). 
Jn 
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On the other hand, by the weak semi-continuous of a norm (or Fatou’s Lemma), we 
have 

[ (|VueP- f |Vuep)dx < liminf / -f |Vu„,ep)dx = S'® « (O). 

JQ n^oo 

Hence, {ue,Ve) is a minimizer of S® p Without loss of generality, we may 

assume that ^ > 0 and £ > 0 for all n. Then since ^ —>■ u^,Vn^E 

in 0, we obtain that Ue > 0, Ug > 0 a.e. in 0. There exists some Lagrange multiplier 
A G R such that 

J-Aug = A^Aag(x)|-«gp*(®)“^Ug + Kaag(x)|itg|““^Ug|ug|^^ in 0, 

|-Aug = A^/rag(a;)|ugp*(®)“^Ug + K/3ag(a;)|Mg|“|ug|^“^Ug^ in O. 

Testing by (ug, Ug), we obtain that A = p ^ D 

Lemma 7.7. For e € (0, s), assume (ug, Ug) is a solution to (I7.211 i given by Lemma 
17.61 Then: 

(i) The family (ug, Vg) is bounded in 

(ii) Up to a subsequence, we have Ug ^ u,Vg ^ v in £)g’^(f2) and Ug — >■ u, Ug — >■ u 
a.e. in LI as e ^ 0. 


(Hi) (u, v) given in (ii) weakly solves 

—Au = Sct,/3,A,/i(Al)fAao(x)|up + Kaao(a;)|u|““^u|t;|^^ /« 0, 

< -Au = SQ,,; 3 ,A,/i(f^)^Atao(a;)|z;p*(®)“^t; + K;9ao(a;)|M|“|t;|^“^t;^ inVL, 

u > 0, u > 0, (u, u) G 

(7.22) 


(iv) If{u,v) f- (0,0), then 

[ ao(a;)[A|M|^*(®) +/i|t;|^*(®) +2*(s)K|ur|u|^]dx = 1 

Ja 

and Ug ^ u and Ug —>■ u strongly in iAQ’^(O). Moreover, (u, v) is an extremal 
function of Sa^p^x,tj.{^)- 

Proof, (i) follows by Lemma lTAl and (ii) is trivial; 

{Hi) Without loss of generality, we assume that £fe 0 as fc —?► oo. For any 
(j> G C^{Ll),tjj G C))°(0), since {ug^.,Vg^f) is a solution of (17.21b with e = Sk, we 
have 


/ (Vitg^ • V(/i + Vugj^ • VV')da; 

JQ 

= 'S'at/3,A,M(^) (Aag(a;)|ugp*(®)"^Ug()) + Kaag{x)\ug 

+ (^Ha6{x)\vgf ^'''>~'^Vgtlj + KPag{x)\ug\°‘\vgf~'^Vg'f'^dx. 


Ugcilugr rfa; 


)‘ 


(7.23) 


43 







Recalling that (x) ao(x) in L°°{sppt{(j))Llsppt{ij;)) and Sa,p,x,fj.{^) 

as fc —?► c», we obtain that 


/ (Vu • + Vx • V'0)dx 

Jn 

= 5'a,/3,A,/i(f^) J (^Aao(x)|Mp ^'''>~^u(j)+ Kaao{x)\u\°'~‘^u(l)\v\^'^dx 

+ Sa,i3,\,f,i^) J (^pao{x)\v\‘^ + Kf3aoix)\u\°^\vf~^v'ijj^dx. (7.24) 


Since (</>, ip) is arbitrary, we see that {u, v) weakly solve (17.221) . 
(iv) By Patou’s lemma, we have 

f ao(x)[A|Mp + 2*{s)K\u\°'\vf]dx 


< lim inf 

k—>oo 


,{x) [A|r 


|2*(s) 


+ p\i 


|2*(s) 


+ 2*(s) 


K Up J W, 


f]dx = 1. (7.25) 


If ao(x) [A|up*(®) + + 2*(s)K|u|“|t;|^] dx ^ 1, since (u, v) ^ (0,0), we 

have 


0< / ao(x)[A|up + 2*(s)/c|u|“|t;|^]dx < 1. 

Ja 

Hence, by (in) and the definition of we have 


^_ In + \^v\^)dx _ 

ao(x) [A|mP*(®) + + 2*(s)At|u|“|t;|^] dx 

/n 

^ 2 

(/q ao(3^)[A|uP*(®) + + 2*(s)/t|M|“|r;|^]dx^ ^ 

>Sa,/3,x,^m, (7.26) 


a contradiction. Hence, ao(x) [A|up*(®) + 2*(s)K|u|“|t;|^]dx = 1. It 

follows that 

[ (|Vuej2 + |v^^j2^^ f (|Vu|2 + |Vu|2)dx, (7.27) 

JQ J Cl 

which implies that Ue^ —>■ u, Ve^ —>■ v inZ)g’^(f2). □ 


7.2 Pohozaev Identity and the Proof of Theorem [TJ] 

Proposition 7.3. Let {u, v) € iAg’^(H) x Dl'^{fl) be a solution of the system 

\-Au =Gu{x,u,v), 

I — Au =Gv{x,u,v), 
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where 


G{x,u,v) = / Gs{x, s,v)ds + G{x,0,v) = / Gt{x,u,t)dt + G{x,u,0) 

Jo Jo 

is such that G(x, 0,0) = 0, G(-, «(•), 'i'(’)) that xtGx^ {■,u(-),v{-)) are in L^(r2). 
Then (m, v) satisfies: 


/ |V(u, v)fx ■ vda 

Jon 

N 

=2N j G{x,u,v)dx + 2 '^^ I XiGxi{x,u,v)dx — {N — 2) j \V{u,v)\'^dx, 

«-'f2 •_I'-'O »-'f2 


(7.28) 


where fl is a regular domain in v denotes the unitary exterior normal vector to 
dfl and |V(u,?;)p := |Vup + jVwp. Moreover, iffl = R^ ora cone, then 



G{x, u, v)dx + 2 


N 

E 

i=l ' 


XiGxi {x, u, v)dx 


{N-2) [ \V{u,v)\^dx. (7.29) 

Jn 


Proof. Since (m, v) is a solution, then we have 

0 = ( — Au + Gu{x, u, v))x ■ Vm = ( — Av + Gy{x, u, v))x ■ Vv. (7.30) 
It is clear that 


JVm|2 ^-2,^ ,2 


—Aux ■ Vm = —div{Vux ■ Vu — x -—^^ 


|Vw| 


—Avx ■ Vv = —div(Vvx ■ Vv — a;-E!L) _ E_?|v^|2^ 

Gu{x, u, v)x ■ Vu + Gy{x, u, v)x ■ Vv 

N 

= div{xG{x, u, v)) — NG{x, u, v) — XiGx^ (x, u, v). 

i=l 

Integrating by parts, we obtain 

r iVuP iVuP 

/ (tTG(CT, u, v) + {Vua ■ Vu — a -—-—) + {Vva ■ Vv — a -—-—)) • I'da 
Jon 2 2 

= [ (nG{x,u,v)- ^ ^ |V(h,u)P + Y^ XiGxi{x,u,v))dx. (7.31) 

Jn ^ ^ i=i ^ 

When M = u = 0 on dfl, we have 


Vu = Vu ■ vu, Vv = Vv ■ vv. 


(7.32) 
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Then by (17.31b . it follows that 

I (^G{a,u,v) + ^\V{u,v)\'^)a ■ vda 

Jon 2 

N 

f N — ‘I 

= / {NG{x,u,v) - - — \V{u,v)\'^+ '^x^Gx,{x,u,v))dx. (7.33) 

Jq ^ .-—I 


i=l 


Moreover, since G{x, 0, 0) = 0, if m = ?; = 0 on 917, we have Gia, u,v) = 0 on 917, 
then we obtain that 

N 

If f N — 2 ^ 

- / \V{u,v)\'^cr-vda = / (^NG{x,u,v) --—|V(m, t;)p+y^ XjGxj (x, u, v))dx 

2 Jon Jq, 2 

(7.34) 

which is equivalent to ( 17.28b . Using polar coordinate transformation, since |V(u, ?;)| G 
we have 

/ \\7{u,v)fdx= \Vuf + \\7vfdx 

Jr^ Jr^ 

= f f \Vu{r,0)\'^ + \Vv{r,9)\'^d9r^~^dr 

Jo JdB^{0) 

pCO 

= / C,{r)dr < oo, 

Jo 


where ({r) := |VM(r,+ \Vv{r,d)\'^d9r^ ^ > 0, then by the absolute 

continuity, there exists —>■ oo such that Ci^n) 0. Since N > 3, we have 


/aB,„(o) 


(|Vu(r„,6»)p + |Vu(r„,6>)p)r„d6» 0, 


(7.35) 


which implies that 

/ |V(m, u)|^cr • i/dcr—>■ 0 as i?—7> oo. 

JdBaiO) 

Since \G{x, u., z;)| G then by the the similar arguments, we obtain that 

/ G{a,u,v)a ■ vda ^ 0 as R ^ oo. (7.36) 

JdBniO) 

When considering 17 = 73^(0) in ( 17.31b . it follows that a ■ v = \x\, 0< 

(Vwcr • Vu) • v < |Vitp(T • v. Hence by (17.35b . we have 

r iVitP 

/ (Vitcr • Vu — cr-i—-—) • —7> 0 as i? —>•+ 00 . (7.37) 

JaBn(o) 2 


>dBR{0) 

Similarly, we also have 


r iVuP 

/ (Vucr • Vv — cr-4——) • vda —>■ 0 as i? —>• +oo. 
JdBniO) 2 


(7.38) 


46 














Finally, by the Lebesgue dominated convergence theorem, when R ^ c», we have that 


'Bh(O) 


|V(u, —>■ / \'\/{u,v)\‘^dx, G{x,u,v)dx ^ / G{x,u,v)dx 

Js.^ Jbr{0) Jk.^ 


and 


N . 


> / XiGxi{x,u,v)dx / 

.-1 JBri(O) 7 r 




XiGxiix, u, v)dx. 


Combining with these results and (17.31b . (17.36b . (17.37b . (17.38b we obtain (17.29b . The 
case of that fl is a cone, we have x ■ i/ = 0 for x € 917, then ( 17.29b follows by (17.28b 
easily. □ 


Corollary 7.1. Let 0 < e < s < 2 and ae{x) be defined by (17.13b . Suppose that 
a>l,/3>l,a + /3 = 2*(s) and 12 is a cone. Then any solution (u, v) of 


—Am = ^ ^ ^(12 )^Aa£(a;)|up*^®) ‘^u + Kaai;{x)\u\°' infl, 

-Av = S^^p^^^^{Ll){^pasix)\v\'^*^'''>~^v + KPas{x)\u\°'\vf~^vj in 12, 

u > 0, t; > 0, (u, u) £ .^ := Dg'^iTl) x I2 q’^(12), 


(7.39) 


satisfies 


-as{x)\uf +-^^^ae{x)\vf + 2*{s)Kae{x)\u\°‘\vf]dx 


JanBi 2*(s) 2*(s) 

= / + -^ae(x)|t;p*(") + 2* {s)Kae{x)\u\°^\vf]dx. 

JQnB^ ^ 

(7.40) 


Proof. Let 


G{x,u,v) := 5'^_^_;,_^(12)[^j^ae(a;)|up*('*^^^ae(x)|z;p*('*)+KaE(a;)|u|“|v|'^]. 

(7.41) 

Noting that 


® /1 


-(s - for |x| < 1, 

_-(s + e)i^pr2+7Xi for |x| > 1, 


(7.42) 
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we have 


Xi ■ Gx,ix,u,v) 

= + fi\unvf]x~a,{x) 


(7.43) 


= 


if |a;| < 1, 

-{s + + ^l^rkl'^] \^ \ X-^+e x‘i 

if |a;| > 1. 


|2*(s) 


Hence, by Proposition |73] we have 

-2{N - s) 

+ + Kae{x)\u\°‘\v\l^]dx 

+ {N -2) [ (|VmP + \Vv\^)dx 

Jn 

= 2£ / 

JnnBi ^ IS) 

ae{x)\v\'^ + Kae{x)\u\°‘\v\^]dx 


2*(s) 


-2e/ ^ae(a:)|u|2*('’) 

JnnB^ ^ IS) 

+ + K,ae{x)\u\°‘\v\^]dx. 


(7.44) 


(7.45) 


(7.46) 

(7.47) 

(7.48) 


On the other hand, since (u, v) is a solution, we have 


/ (iVup + |Vt;p)da; 

Jn 

= /3 A/.(^) [ [Aae(a;)|up*^®^ +^Oe(a;)|t;|^*(®) + 2*{s)Kaeix)\u\°‘\vf]dx. 

Jn 

(7.49) 

Recalling that ^ ^ ^{fl) > 0, e > 0, by (17.451) and ( 17.491 ). we obtain (17.40b . □ 

Corollary 7.2. Let 0 < e < s < 2 and ae{x) be defined by (17.13b . Suppose that 
a > l,/3 > l,a + /3 = 2*(s) and LI is a cone. Let (ue,Ve) be a solution to (17.21b 
given by Lemma \7.6\ Then up to a subsequence, there exists some (u, v) € such that 
Ue —>■ u, Ue —?► u strongly in i9g’^(0) ai e —>■ 0. 

Proof. By Lemma lTTTl we only need to prove that (u, v) ^ (0, 0). Now,we proceed by 
contradiction. We assume that tt = u = 0. Let x(x) G (O) be a cut-off function 
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such that x(x) = 1 in Br. fl fi, x(x) = 0 in il\Br, recalling the Rellich-Kondrachov 
compact theorem and 2 < 2 *(s) < 2 * := we have Ue —0 in for all 

1 < f < 2* if 0 ^ Oi. Hence, it is easy to see that 


/ [Aae(a;)|xw£p*^'’^ ++ 2 *{s)Kae{x)\xUe\°‘\xVef]dx 
Jn 

= f [Aae(a;)|uep*<^'’) + nae{x)\ve\'^'‘^'''> + 2*(s)Kae(a;)|ue|“|t;e|^]dx + o(l) 

JflnBr- 

=:r]r + o{l). (7.50) 

On the other hand, by the triangle inequality, we have 

[ (|V(xUe)P + |V(xUe)P)dx 

Jn 

= I (l(Vx)Me+ xVM£p + |(Vx)ue+xVu£p)dx 
Jn 

\xf\Vu,\^dxy 


\{Vx)\^uldxy 

\{yx}{^v^dx}^ 


n 


n 


IxHVuepdx)' 


= / (|VWeP + |VUe|2)dx + o(l) 

JnnBr 

:=ar + o(l). 

By (17.50b and l7.51l we obtain that 

5'S./3,A.M(^)(7r +0(1))^ <ar + 0{l). 


(7.51) 


(7.52) 


Similarly, we take x{^) ^ (O) such that x{^) = 0 in H O and x = 1 in fl\B 2 r- 

Then by repeating the above steps, we obtain that 

S^,p,xJ^){l-Vr+o{l))^ <Slp^^^^-ar + o{l). (7.53) 

By (fOa and (173^ . we deduce that 

(p, + o(l)) ^ + (1 - 77 , + o(l)) ^ < 1. (7.54) 

Notice that h{t) := <=*(=) + (1 — t) satisfying that minjgjQ h{t) = 1 and only 
achieved by f = 0 or f = 1. Hence, we obtain that 77 , = 0 or 77 , = 1 for any r > 0. 

But by Corollarv l7.ll for any e G (0, s), we have 


/nnBi 


[Aae(a;)|'U£p*(®) + /j,ae(x)lvsl^*^'’^ + 2* {s)Kaeix)\us\°‘\vef]dx 


f [Aae(a;)|7te|^ + /iag(a;)|7;e|^ + 2*{s)Kae{x)\ue\°‘\vEf]dx. 

JnnB^ 
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Combined with the fact of that 


/nnsi 


[Aa£(a;)|MeP + fiae{x)\Ve\‘^ + 2*{s)Kaeix)\Ue\°‘\Vef]dx 


+ 


JnnBf 

we obtain that 


[Xae{x)\Ue\'^ + Hae{x)\Ve\'^ ^’''>+2*{s)Kaeix)\Ue\°'\Vef]dx = l, 


innBi 


[Aae(a:)|U( 


|2*(s) 


/nnsf 


[Aae(x)|i 


|2*(s) 


+ fiaeix)\Ve\^'‘^‘'^ 


2* {s)Kae{x)\Ue\°'\Ve\^]dx 

+ 2*{s)Kae{x)\Ue\°‘\Vef]dx 


_ 1 
“ 2 ' 

Hence, we have rjr = \ for any r > 0, a contradiction. 


(7.55) 

□ 


Proof of Theorem l7.lt Let be a solution to (I7.211 i with e = Sk given by 

Lemma 1731 and efc —>■ 0 as fc —>■ +oo. Up to a subsequence, we may assume that 
Uej, ^ u,Vek ^ u in and ^ u,Vc ^ v a.e. in H (see Lemma ITtT i. 

Then if 0 ^ H, by Rellich-Kondrachov compact theorem, it is easy to see that —?> 
u, Vet, ^ Strongly in Dq'^{Q). When H is a cone, by Corollary 17.21 we also obtain 
that Ue —?► UjWfc V strongly in £)Q’^(fl). By (iv) of Lemma 17.71 we obtain that 
{u, v) is an extremal of Sa,i 3 ,x,ij,{Xl), the proof is completed. □ 

Remark 7.3. When H is a cone, let (u, v) be the extremal obtained as limit of{ue,,, Ve,,), 
the solution to (17.211 1 with e = Sk given by Lemma \7^ Then by Lemma !Z7l Corollary 
\7.2\ and the formula (I7.551 l. we see that {u, v) satisfies 



pao(a;)|up + 2*(s)Kao(a;)|M|“|u|^]da; 

+ /rao(x)|up + 2*{s)K,a[){x)\u\°‘\v\^\dx 


1 

2' 


Such a property has been observed for the scalar equation. 


7.3 Existence of positive ground state solutions 

_ 2 

By (17.61) . we always have Sa,i 3 ,x,tj,{Xl) < (max{A,/r}) p,s(LI). When H is a 
cone and s € (0, 2),a > l,/3 > l,a + /3 = 2*(s), by Theorem |7.II Sa,/3,x,fj.i^) 
always attained (although the extremals may be semi-trivial). Indeed, if Sa,i3,x,tJ.{^) = 


50 








(inax{A, fi}) then can be achieved by semi-trivial function. 

To see this, we just need to plug in the pairs (C/,0) or (0, C/), where U is an ex¬ 
tremal function of But, under some special conditions, can also 

’ ' 2 

be achieved by nontrivial function even A, = (max{A,//}) see 

Theorem l7.4l below. However, if 


Sa,p,x,iJ.i^) < (max{A,/r}) 


(7.56) 


then the extremal functions (hence the positive ground state of the system (17.11) 1 of 
Sa,i3,x,tJ.{^) must be nontrivial. Therefore, next we need to search some sufficient 
conditions to ensure the above strict inequality (17.561) . We obtain the following theorem 
on the existence, regularity and decay estimate. 

Theorem 7.2. Let VL be a cone in (especially, Q = or LI = ) or LI be an 

open domain but 0 ^ LI. Assume s G (0, 2), k > 0, a > 1, /3 > 1, a -|- /3 = 2*(s). 
Then system (ED possesses a positive ground state solution [(j), ip) (i.e., (p > > D) 

provided that one of the following conditions holds: 


(oi) X > p and either 1 < (3 < 2 or 


(3 = 2 

K > 


(02) X = p and either min{a, (3} <2 or 


2*(s) 

min{a, /?} = 2, 

K> ^ 


2*(s) 


(03) X < p and either 1 < a < 2 or 


a = 2 

K > 


2*(s) 


Moreover, when LI is a cone, we have the following regularity and decay properties: 

( 61 ) ifO<s<^,f,i;GCHn); 

( 62 ) ifs = (p,ip G for alio < j < 1 ; 

(63) ifs > (p,'tp G 0^’"^ {LI) for all 0 < 7 < ■ 

When Lt is a cone with 0 G dLl (e.g., O = R^), then there exists a constant C such that 

\<P{x)\, \P;{x)\ < C{1 + \^cp{x)\, |Vr/>(x)| < C\x\-^. 

When LI = R^, 

\(p{x)\, \ip{x)\ < C(1 -f |a;|"^), |V<()(a:)|, \^'ip{x)\ < C\x\~^~'^ 

In particular, if LI = M.^, then {(p{x),'ip{x)) is axially symmetric with respect to the 
xjY-axis, i.e., 

{(p{x),ip{x)) = {(P{x',xn),iP{x',xn)) = (<()(|a;'|,a:Ar),'i/'(|a:'|,a;Ar)). 
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Remark 7.4. The conditions (ai) — ( 03 ) imposed in Theorem U .2\ are some sufficient 
conditions to ensure the inequality (17.561) (see Lemma \7. 91 below ). But they are not 

necessary conditions. For example, when X > p and 1 < a < 2, we can not exclude 

2 _ 

that < X 2 *(s) ps{^), i s., (17.561 ) may be true. 

Define the functional 

f (|VmP + |Vt;p)(ix 
2 Jq 

~^)LW ^ ^ 2ffis)n\unvf] dx (7.57) 

and the corresponding Nehari manifold 

M := {(m, v) G ^\{0, 0} : J(u, v) = 0} (7.58) 


where 


J(u, v) 

■■= (^'(u,v),(u,v)} 


= [ (IVtrp + IVuHdx- [ ^[AH2*(*)+;i|z;f(«) 

Jq Jq fI 


(7.59) 


2*{s)K\u\‘^\vf]dx. 

(7.60) 


By Lemma l4n is well defined. Define 


Co := inf $(u, v), 

then basing on the results of Section 4 and Section 6, we see that 


(7.61) 


0 < Co < [i - " (max{A,/r}) =. (7.62) 

Moreover, we have the following result. 

Lemma 7.8. Let LI be a cone o/K^ or LI be an open domain but 0 ^ LI. Assume that 
K > 0, s € (0, 2), a > 1, /3 > 1, a + /3 = 2*(s) and let cq be defined by (17.611) . then 
we have 

1 1 ,, 

Co < b - 


2 *(s)^ 


if and only if 


Proof. A direct computation shows that 

pi 1 

^2 2*(s) 


]{s^,p,xA^)y 


x,p}} 

(7.63) 


(7.64) 

2*(s) 


:*{s)-2 ^ 

(7.65) 


□ 
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Then combining with the conclusions of Section 6 , we have the following result; 

Lemma 7.9. Let Q be a cone in (especially, Q = and O = ) or LI be an 

open domain but 0 ^ LI. Suppose s € (0, 2),k > 0, a > 1,/? > l,a + /3 = 2*(s). 

Then Sa,p,\,^(Ll) < (max{A,/r}) p^siLl) if one of the following holds: 


(i) X > fi and either 1 < (3 < 2 or 


(3 = 2 

K > 


2*(s) 


(ii) X = fj, and either min{a, (3} < 2 or 


min{a, /?} = 2 , 

A 


K, > 


2*(s) 


iiil) X < u and either 1 < a < 2 or < 

b> 2 ^ 

Proof It follows by Corollary 16.31 Corollarv l6.4l Lemma lOl and Lemma 177^ □ 


Proof of Theorem 17.21 Under the assumptions of Theorem 17.21 firstly by Theorem 
17.11 Sa,i 3 ,\,fi{Ll) is attained by some nonnegative pair (u, v) such that (u, v) (0, 0). 
On the other hand, by Lemma 177^ we have 

- 5 'a./ 3 ,A.M(^) < (max{A,/i}) p.s{Ll). 

Hence, we see that u 0,v 0. Hence, by Proposition !?.!! 

is a ground state solution of system (17.1b . Then by the strong maximum principle, it 
is easy to see that f > 0,, ip > 0 in Lt. We note that the arguments in Proposition l3.ll 
and Proposition 13.21 are valid for general cone. Combining with Proposition 13.31 we 
complete the proof. □ 


7.4 Uniqueness and Nonexistence of positive ground state solutions 

In the previous subsection, in Theorem 17.21 we have established the existence of the 
positive ground state solution to the system dTTTT i. Now, in the current subsection, we 
obtain the uniqueness of the positive ground state solution to the system (17.1b . Define 


G{u, v) 


In + |Vt;p)da; 


(/n{A 






2*(s)k 


\u\°‘\v\t 

1x1“ 


■)c?a 


2*C«) 


then we have 




inf G{u,v). 

(«,x)G®\{(0,0)} 


{u,v) ^ (0,0) 


(7.66) 

(7.67) 
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For any u 0,v 0 and t > 0, we have 

J^{\yu\^ + \yv\H^)dx 


G(u, tv) = 


Hence, 

G(m, tu) = 


/n + |Vupt^)dx 


2 ^ 


-. (7.68) 


(/„ (A^ + 

l+t^ J^\\7u\‘^dx 

[A + + 2 *{s)ntP] ^ 


2 ^ 


■=9it)- 


fn iVMpdx 


_ 2 

We define g(+cxD) = lim g{t) = fi 2 *(«), then we see that 

t—¥-\-oo 


(7.69) 


/n iVtApdx 


G(0,t;)= lim G(t;, to) = ( 7 (+oo)- 

t^+oo /, ^ 




Hence, we have 


S'a,/3.A.M(^) = inf G{u,v) 

(ii,j;)G®\{(0,0)} 

< inf inf G(u, tu) 
ueDg’'^{n) te[o,+oo) 


= inf git) 

[0,+oo) 


fn iVupdx 


inf 

ueDi’^(n)\{o} t r 

\ t) £*2 13 ^ I ® I 


= inf g{t)g.s{^). 

iG[0,+oo) 


Moreover, we can obtain the follow precise result: 
Lemma 7.10. S'q fl A u(f 7 ) = inf git)g,si^), where 

' ’ ’ tG[0.+oo) 


(7.70) 


ait) ■= 


i + t^ 

[a + ^f2*(s) _|_ 2*(s)Kf^] 


(7.71) 


Proof. By (I7.701 i. we only need to prove the reverse inequality. Now, let {(unton)} 
be a minimizing sequence of Sa,p,\,)j,{^). Since G{u,v) = G(to, to) for all f > 0, 
without loss of generality, we may assume that 























and 


G{Um Vji) — *S^a,/3,A,/i(^) ~t“ ^^(1)- 


r u |2*(s) 

Case 1: liminf / —;- dx = 0. Since {wnl is bounded in by 

n^+oo/Q |a;|® 1^1 


Holder inequality, up to a subseqeunce, we see that 


| 2 *M b„| 2 *('> 


+fi —j—j--|-2 *(s)k 


Un Vn 




)dx = n I 

Jn 


|2*(s) 


-da;+o(l) = /r+o(l). 


In \x[ 

Hence, 

lim G{u,Vn) > lim G'(0,t;„). 

n—^oo n—>-oo 

We see that (0, u„) is also a minimizing sequence of Hence, it is easy to 

see that 

=p(+oo)^s(H) 


> inf g{t)n,{n). 

tG(0,+oo) 

|^^| 2 *(s) 

Case 2: liminf / " —dx = 0. Similarly to Case 1, we can obtain that 


(7.72) 


n—>-+oo 


n 


X' 


Sc.,p,xA^) =A-5^ps(H) 

=g{0)Hs{n) 

> inf g{t)fj.,{n). 

t£[0,-\-oo) 


(7.73) 


Case 3: Up to a subseqeuce if necessary, we may assume that lim 

\x\' 

| 2 *(s) 

-dx = 1 — (5 > 0. Let tn > 0 such that 


|2*(s) 


-dx = 


(5 > 0 and lim 


n->.+coJ^ \X 


| 2 *(s) 


-dx = 




|2*(s) 


-dx. 


Jn ur Jn Fr 

then we see that {f„} is bounded and away from 0. Up to a subsequence, we may 

1 

assume that tn ^ to = (t^) ■ Now let Wn = '^Vn, then we have 


|2*(s) 


Jn Fr 

and by Young’s inequality, we have 


-dx = 


\w. 


|2*(s) 


-dx 


In FI' 


(7.74) 


a f 13 |w„| 2 *(s) 

-dx < 


In FI' 


2*(s) Jn |a;|' 


In Fr 


-dx = 


dx 3- , , 

2 *(s) . 

Jn kh 


-dx 


In Fr 

dx. 


(7.75) 
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Hence, 




Jq \yUn\'^dx 


{In + + 2 -(»)k(„^ 


_L „^2*(s) KJ^ 9*r.V+/3 ^ 

J^tl\\/Wn\^dx 


( f I ,2*(s) |u;„|2*('i) o*f„),.f 

--[jp- hZ[S)Ktn -^ 

1 


>- 


0 IWnMlUrJfA ^ 




/q iVwnpda; 


A + ntn + 2*{s)Kti dxj 

In iVwnpdx 

knP*M 


I to ^ L ' ' 

A + + 2*{s)Ktn ^ Jq ^dUlj^^dx'j 


>- 


1 


A + fltn ^ ^ + 2*{s')Ktn 

'^n 


2 *(„) 


-Ps(rj) 


+ 


A + ^itn + 2*{s)Ktn '' 
~9{^n)lls (^)- 

Let n —>■ + 00 , we obtain that 


~ fis (A^) 


(7.76) 


Sa,i3,x,tJ.{^) > g{to)9s{^) > inf g{t)fis{^)- 

te(o.+oo) 

Thereby Sa a \ n{^) = inf g(t)/is(Ai) is proved. □ 

’ ’ ’ tG(0.+oo) 

Basing on Lemma 17.101 we can propose the “uniqueness” type result as following: 

Theorem 7.3. Let fl either be a cone in M.^(in particular, 17 = and 17 = ) or LI 

be an open domain but 0 ^ 17. Assume s £ (0, 2),k> 0, a>l,/3>l,a + /3 = 2*{s). 
Let (</), tp) be a positive ground state solution to problem (17. Il l, then 


(j) = C{to)U, ip = toC{to)U, 


where U is the ground state solution of 

f-Au = in 17, 

1 u = 0 on 917, 


while to > 0 satisfies that 


9 (to) 


inf 

tG(0,+oo) 


9it) 


(7.77) 


(7.78) 
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and 


25 ^ 


c{h) := ^ , , (7.79) 


^ A “t" ^ ^ “t“ 


where g(t) is defined in ( I7.77I ). 

Proof. By the processes of Case 3 in the proof of Lemma [7.101 if Sa,p,\,fj,{^) is 
attained by some nontrivial function {u,v), i.e., u 0,v 0, then there exists 

some fg > 0 such that v = t^u, where u is a minimizer of and fg satisfies 

g{to)= inf g{t). 

iG(0,+oo) 

Now assume that u = CU, v = t^CU, then a direct computation shows that 

/ +2*(s)/t|ur|u|^]dx= 1 

Jo. Fl 


if and only if 

c l 

\A + /rfg + 2*{s)Ktg j 

Finally, we see that </> = [5'a,/3.A./j(f^)] u,ip = [5'o,,/3,A,/i(Al)] 2*('»)-2 v, we com¬ 
plete the proof. □ 

Remark 7.5. Under the assumption that 

Sa,i3,x,^t{^) < (max{A, p}) ps{^), 

we have seen that the problem dTTT i possesses a positive ground state solution. But 

the converse usually is not true. Next, we construct an example where = 

_ 2 

(max{A, p}) p.s (Al) but the system (17. Il l still has (multiple) positive ground state 
solutions. 


Theorem 7.4. Let LI be a cone in or LI be an open domain but 0 ^ LI. Assume the 
following conditions hold: 

(a) 0 < s < 2, 

fa = 2 

(b) either a > 2 or < 

\g>2K 


(c) either j3 > 2 or 


/3 = 2 
A > 2k 


(d) a + ^ = 2*{s). 


57 











Then we have one of the following conclusion: 

(I) If s = 1, a = /3 = 2, A = // = 2 k > 0, then the set of all extremal functions of 
is given by 

A := t2U) : > 0, ^2 > 0, (^ 1 ,^ 2 ) ^ (0, 0) and 

U is an extremal of iXs{Tl) k (7.80) 


(2) Except for the item (1) above, has no nontrivial extremal function. 

Remark 7.6. Under the hypotheses {a)-{d), the dimension of the space has to be 
three. Therefore, we can only establish the above theorem in 

Proof. We proceed by contradiction. Assume that Sa,p,x,ti{^) has a nontrivial ex¬ 
tremal (it, v), then by Lemma 17. 101 (see case 3 of the proof), we see that there exists 
some fp > 0 such that v = tgu and u is an extremal of /is(fl). Moreover, to attains 
the minimum of g{t), where g{t) is introduced in ( 17.71b . By conditions (6) and (c), 
we see that g''{0) > 0 and g'(f) < 0 for t large enough. Hence, {f > 0 : g'{t) = 0} 
has at least 3 solutions {ti,t 2 ,to} such that 0 < ti < t 2 = to < to < 00. A direct 
computation shows that 


, -2pf2*(^*)-i + 2KatP+^ - 2K/3f'5-i + 2\t 

g [t) = - 2 - 

[a + + 2*(s)Kf/^] 

=-- 2 - 7 + Kf3t^~^ 

[a + + 2*(s)Kf^] 


A). 

(7.81) 


Define 

h{t) := pt^ — A, (7.82) 

then we obtain that {f > 0 ; h{f) = 0} has at least 3 solutions {ti,t 2 ,to} such that 
0 < ti < t2 = to < to < 00. 


Case 1:P = 2 and 2k — A < 0. For this case, h{t) = pt^ (®)“2 _ ^^^2 _^2k — \. By 
Rolle’s mean value theorem, {f > 0 : h'{t) = 0} has at least two solutions fi, f 2 such 
that 

ti < ii < t2 = to < i2 < to. 


Note that 


{f > 0 : hft) =0} = {t>0: p[2*{s) - 2]f2*(«)-4 _ 2Ka = 0}. 


In particular, the set {f > 0 : p[2*{s) — 2]f^ ^ — 2Ka = 0} has a unique solution 

if 2*(s) 7 ^: 4, a contradiction. Hence, 2*(s) = 4 and p = 2k. Recalling that 2*(s) = 
'^^N- 2 '* , s € (0, 2), we obtain that s = 1, a = 2*(s) — j3 = 2. Then 


5 (^ 0 ) 


_ 1 + ^0 _ 

[A + 2Kto + dKfpJS 


1 


(7.83) 
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which implies that 


X = fi = 2 k. 

It follows that g{t) = Hence, when iV = 3, s = 1, a = /3 = 2, A = p = 2k, the 
extremals of are given by (I7.801 l. In particular. 




2k(1 + 


{U, tU):t> 0} 


are all the ground state solutions of 


f-Au - 2k'X^ - 2k^ in 

< -Av - 2k^ = 2 k§ in H, (7.84) 

> 0,M,U G £>q’^(H), 

where U is the ground state solution of (17.77b . 

Case 2: P > 2. For this case, h{t) = — Kat^ + — A, similarly we 

see that the equation h{t) = 0 (f > 0) has at least three roots ti < t 2 = to < It 
follows that {hpt), f > 0} has at least two roots ii and ^ 2 , which implies thatp(f) = 0 
{t > 0) has at least two solutions. Where p{t) is defined by 

p{t) := p[2*{s) - 2]f2*(«)-/3 _ KaPt^ + kP{P - 2). (7.85) 

A direct computation shows that p”{t) > 0 when a > 2. Hence p{t) = 0 (f > 0) 
could not have more than one solution, a contradiction. If P > 2, a = 2, we have 
p(t) = [p[2*{s)-2] — KaP] + kP{P — 2), which also has at most one positive root, 
a contradiction too. 

We note that for the case of /i > A, we will take 

g{t) := gi-) = -^ 

^ [p + + 2*{s)Kt°‘] 

and the arguments above can repeated (P is replaced by a now). We complete the 
proof. □ 

Corollary 7.3. Under the assumptions of Theorem \7.4\ there must hold 

= (max{A, /r}) ps{^)- 

Proof For the special case 7V = 3, s = l,a = /3 = 2, A = /r = 2k, a direct compu- 

2 

tation can deduce it. For the other cases, if (max{A, p}) 

_ 2 

then there must hold that < (max{A, p}) /is(H). By Theorem 17.II 

and Lemma ITU »S'a,/3,A,/i(^7) can be achieved by some nontrivial extremal {u,v), a 
contradiction to Theorem l7.4l □ 
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7.5 Further results about cones 


Assume that 0 < s < 2, a > l,/3 > 1 ,q: + /3 = 2*(s). Based on the results of 
Theorem 17.II we see that when 17 is a cone, is always achieved. In this 

subsection, we always assume that 17 is a cone. We shall investigate more properties 
about in terms of 17. Let us begin with a remark. 

Remark 7.7. Assume that 17i, 172 domains o/M^ and 17i C 172, then it is easy to 
see that Z7q’^(17i) C Z7 g’^(172). Then by the definition of (see the formula 

(Oa we see that 

Lemma 7.11. Let {17„} be a sequence of cones. 

(i) Assume {17„} is an increasing sequence, i.e., Lin Q 17n+i, then 
lim (A7n) = ‘S'cK,/3,A,/i( lim 17yj) = 

n—foo n—foo 

where 

OO 

17 = lim 17„ =11 17„. 

n—¥(X> 

n—1 


(ii) Assume {r^n} ^ decreasing sequence, i.e., 3 then 

lim. Sql,^ — Set ^ X lim ^n) — 

n—foo n—foo 


where 

OO 

17= fl 17„, 

n—1 

and we denote that Sa,p,x,^(^) = +oo if meas(VL) = 0. 

Proof (i) By Remark lLTl we see that {>S'a^/3_A,/j(^2„)} is a decreasing nonnegative se¬ 
quence. Hence lim„_>.oo Sa,p,x,n{^n) exists. Also by Remark lT/Tl and 17 = U^i 2 
17„, V n, we have 

Sa,| 3 ,\,^J.(^) < lim Sa,p,x,fi(,^n)- (7.86) 

n—foo 

On the other hand, for any e > 0, 3 (ue, vf) G C^(17) x C^(Ll) such that 


and 


(A 


/ [|VMe|^ + \WVef]dx < S'a,/3,A./x(^) + ^ 
Ja 

|2 (s) |,,_|2 (s) 


+ 


+ 2*(s)k1 


Fl FI FI 

Then there exists some Nq large enough such that 

MejUg G C^(Ll„) for all n > Nq. 

Hence, by the dehnition of Sa,p,x,^^{^n) again, we have 

^q,/ 3 ,A,^(17n) ^ ^ for all 71 ^ AA). 


)dx = 1. 


(7.87) 


(7.88) 


(7.89) 

(7.90) 
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Let n go to infinity, we have 


n—^co 

By the arbitrariness of e, we have 

lim (7.92) 

n—¥oo 

Now, (17.861 1 and (17.92b say that 

lim — ^a,/3,A,^( lim (7.93) 


(ii) By Remark iLTl we see that {S'a,/ 3 ,A./i(i^ra)} is an increasing sequence. Let us 
denote 

S := lim 


For any n, let {un,Vn) be the extremal function to by Theorem 17.11 We 

can extend u„ and by 0 out side fin. By Remark I tTJI we have (A-^— 

+ 2*(s)KJ^LLrKll)da; = 1 and 




|2*(s) 


|2*(s) 


\'^n\ ' ' It^nl ' ' n*/- ^ Wn\ j 


/ (^- 

/nABi |a^r 


(7.94) 


Case l-TOeas(O) = 0: In this case, we shall prove that S = oo. Now,we proceed by 
contradiction. If5 < oo, {u„}, {u„} areboundedsequences in iAQ’^(ni). Thenuptoa 
subsequence, we may assume that Un ^ u,Vn ^ v in i9Q’^(f2i) and Un ^ u,Vn ^ v 
a.e. in fli. Since rneas{fX^^^ fin) = 0, we get m = 0, v = 0. On the other hand, by 
applying the same argument as Corollary 17.21 we can obtain that Un ^ u,Vn ^ v in 
jl^). Then we have 


/ (A- 

/R-N 1^1 


|2*A) 




(7.95) 


a contradiction. Hence S' = oo. 


Case 2-0 is a cone: In this case, by Theorem l7.ll Sa,/ 3 ,\,fj.{fl) is well defined and can 
be achieved. Notice that for any n, we have ft C 0„, by Remark iTSl again, we have 
Sct,/3,A,/i(Oyi) ^ So,A,A,/i(O). Hence 

5 < So./3.A.;.(0). (7.96) 
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Thus, {un}^ {t'n} are bounded in D^' (fii) for this case. Arguing as before, it is easy 
to see the weak limit tt ^ 0 , u 0 and 


0 < 


|2*(s) 


.|2*(s) 


+ M" 


m Fr I2;i 

We claim that (u, v) weakly solves 


+ 2*{s)t 


M U 


1 ^ 


')dx < 1 . 


(7.97) 


—Am = ^M + Kct-nfr^lMr "m|u 


|a-2„ 


p) in n, 

Am = + /t/3|^|M|“|Mp“^M^ in 

M > 0, M > 0, (m, m) € ^ := DQ'^{n) X Dq'^{D,), 

Since (fl) is dense in i9g’^(0), we only need to prove that 

[ Vm • V(/> + Vm • Vi/' 

hi|2 


(7.98) 


=S 




+ M" 


na- 


|M|“ ‘M(^|mP 


|a — 2 , 


in Fr FI 

for all e c^{n) X c^{n). 


+ k/3 


M M 


P "^vip 


)dx 


(7.99) 


Now, let (()>, 1 /’) G C^(fl) X C^(fi) be fixed. Notice that fi C we have G 
79g’^(nra),V n. Then 

/ Vm„ • V(^ + Vm„ • Vi/i 


-S^,P,xA^n) +p 

, „ l^n | “ | N , 

+ KQ!- —j - !- h up- -—j- dx. 


Since supp{p) supp{ip) C fl, we have 

/ Vm„ • V(/) + Vm„ • Vi/' 

7n„ 

-S^,P,xA^n) J^^[X +P 


(7.100) 


|m„|“ ^m„(/)|m„P „|m„|“|m„P ^Mnl/l 


+ it/5- 


|a;|“ |a:|“ 

Then apply the similar arguments as Corollary 17.21 we have 

/ Vm • V(/) + Vm • Vi/j 


^ dx. 


for all n. 


(7.101) 


in ^ |a;p |a;p 


Aca 


ImT *m(/)|mP 


|Q! — 2 „ 


|a;p 

(7.102) 
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Thereby the claim is proved. By (17.97b and 2 < 2*(s), we have 




<- 


+ \Vv\'^]dx 


(la + 2 . 


<- 


|a:p ^ d- |a;p “r ^ \o)ra |^|3 

/s^[|Vm|2 + \Vv\‘^]dx 


Jq ^ (SjK |^|, )ax 

It follows from (17.96b and (17.103b that 

‘^CK,/3,A,/i (^) — S = lim 


= s. 


(7.103) 


and 


In 


. h| 2 *M 

(.^“FTTT-^ M- 


+ 2-(,K!^!5!!)dx = l. 


|a;p |a;|“ |a;| 

Hence, {u, v) is an extremal function of The proof is completed. □ 


Define 

i^a./ 3 ,A,/i ■= {>S'a,/ 3 ,A,/ 2 (^^) : H is a cone properly contained in R^\{0}}. (7.104) 

For any given unit versor z/ in let 

He := {cc € : a; • 12 > |a;| COS0}, 0 €(O, 7 r]. (7.105) 

Definition 7.1. Assume 1 < p < iV, —oo < t < N — p, we denote by D]’^{rt) the 
completion of C^(fl) with respect to the norm 

ll«lh=(/^^^^)" (7-106) 

Then we have the following result: 

Proposition 7.4. If F is a closed subset of a k— dimensional subspace ofW.^ with k < 
N — t — p, then D\’^{VL) = Dl'^{fl\F). In particular, = 79^’^(R-^\{0}) 

provided N — t — p > 0. 

Proof Without loss of generality, we assume that H = R^. Notice that (R^\F') C 
C^(R^), by the definition, it is easy to see that i9(’^(R^\F) C ’^(R^). 

On the other hand, for any u € 79^there exists a sequence {pn} C C^(R-^) 
such that 

\\pn — u\\^= f rfx —» 0 as n —» OO. (7.107) 

Jrzv \x\^ 

If there exists a subsequence of {puk} such that supp{pn^) H F = 0, then {puk} ^ 
C'“(R^\F), and it follows that u € 79(’^(R'^\F) and the proof is completed. Hence, 
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we may assume that supp{(pn) fl F ^ 0 for any n without loss of generality. Now, 
for any fixed n, we may choose a suitable cutoff function xs such that xs = 0 in 
Fs,xs = 0 in R^\F 2 s,xs e (0,1) in iR^\Fs) n F 2 S, jVxal < where F c 
and 

Fs := {x € : dist{x,F) < <5}. 

We note that xs^n £ C^{R^\F) for all (5 > 0. 

Now, we estimate ||x 5 </^n — 


|V(x5V5n - PnW 


dx 


/RW 


[|V(X5 - + 2V(x5 - 1) • Vpnixs - + {XS “ 1)^|V(^„P] 


< 


r \2{\y{x5-l?pl\ + {xS-l?\y^n\^) 


-dx 


-dx 


/RW 


<2P 


L 


' SUpp{ipn,)r\F2S I® 

:=/ + IF 


\^2^\pPJdx + 2P 


f SUpp{<Pri) 


1X5 


iipr^ 


-dx 


By the Lebesgue’s dominated convergence theorem, it is easy to see that 

II = 2pf 1 x 5 - ir ^ 0 as J ^ 0. (7.108) 

J SUpp{(pn) 1^1 

Recalling that \Vxs\ <h there exists some c„ > 0 independent of <5 such that 

1 = 2^ [ ^^\^?.\dx<Cnd)PS^-^-P. (7.109) 

J SUpp{ipn)nF2S U 

Hence, when k < N — t — p, we also have 

/^0as(5^0. (7.110) 

Hence, we can take some (5„ small enough such that 

||X5„<^n-</?n|r< (7.111) 

n 

Now, we let u„ := xs^^n S C'^(R^\F), we see that ||it„ — u\\p —>■ 0 as n — 
00 . Hence, u € dI’P(R-^\F). Thus, C dI’P(R-^\F). Especially, when 

N — t — p > 0, take /c = 0, we see that dI’P(R^) = Z7h^(R^\{0}) and the proof is 
completed. 

□ 


Lemma 7.12. 

5 'a,/3,A,M(^7r) = Sa,f 3 ,X,f_t(R^) far N >F 
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Proof. Take F = Fn := F n Bn{0). We note that Fn is a closed sub¬ 

set of a 1—dimensional subspace of and lim Fn = F. Then by Proposition 

n—foo 

IT41 Dl’'^{R^\Fn) = for any n. Then it follows that dI’'^{R^\F) = 

Dl'^R^). That is, = dI^\R^). Hence, = Sp,aA^^)- □ 

Theorem 7.5. For every t > ^ p \ p,’ there exists a cone f2 in R^ such that Sa,p,\,p{^) 
T. Moreover, when N > A, we have 5^, p x p — Sa,p,\,p{A^)- 

Proof. Define a mapping r : ( 0 , 7 r] i—>■ ]R_|_ U {0} with t{6) = Sct,p,x,p{^e). Then 
by Remark 17^ we see that the mapping r is decreasing with related to 9. Evidently, 
t{9) is continuous for a.e. 9 G (0, tt]. Furthermore, we can strengthen the conclusion. 
Indeed, let 9 G (0, tt) be fixed. For any f 9, by (i) of Femma l7.1 II we have 

lim T{9n) = T{9). (7.112) 

n—foo 

On the other hand, for any | 9, by (ii) of Femma 17.1 II we also obtain (17.1121 1. 
Hence, r is continuous in (0,7r). In addition, r(0„) f Sa^p^x^p{i}A = §-a,p,x,p 
9n'\ and rifn) t +00 as | 0 - 

Especially, when iV > 4, by Femma l7.12l we have that 

^,P,\,p ~ ^a,P,\,p{^A — )■ 


□ 


7.6 Existsence of infinitely many sign-changing solutions 

In this subsection, we will study the existence of infinitely many sign-changing solu¬ 
tions as an application of Theorem l7.2l 


Theorem 7.6. Assume s G (0,2),k > 0, a > l,/3 > l,a F (3 = 2*(s). Let ilg 
be defined by (17.1051) for some fixed 9 G ( 0 , 7 r]. Suppose that one of the following 
conditions holds: 


(oi) X > p, and either 1 < (3 < 2 or 


(3 = 2 




( 02 ) X = p and either min{a, (3} < 2 or 


2*(s) 

min{a, /?} = 2 , 

K > ^ 


2*(s) 


(03) X < p and either 1 < a < 2 or 
Then the problem 


a = 2 

K > 


2*(s) 


'-Au - Xj^\uA^^^~‘^u = Kaj^\u\°‘~'^u\v\^ in fig, 

< -Av - pj^\vA^'''>~‘^v = K(ij^\u\°‘\v\^~^v in rig, (7.113) 

fu,v) G 2! := oAirig) X 
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possesses a sequence of sign changing solutions {{uk,Vk)} which are distinct under 

Cfc 

the modulo rotations around v. Moreover, their energies Ck satisfies > +c» 


as k ^ oo, where 
1 


2KN-1) 


Cfc • — 7 


' Cte 


[\Vuk\ + \Vvk\ ]dx 


1 


2*(s) 


f I I rn'i'i 


Proof. The idea is inspired by El. We will construct a solution on fig by gluing to¬ 
gether suitable signed solutions corresponding to each sub-cone. Using the spherical 
coordinates, we write = {6i, ■ ■ ■ , 9 m-i : € S'^, i = 1, • • • , — 1}. For any 

fixed fc € N, we set 

and for every choice of (ji, j 2 , • • • , jiv-i) G {0,1,2, • • • , 2^= - 1}^“\ 

-/.r. 


X 

G Sf'i X 


■ |a;| 

3i 

JlU" JiV-1 J 

,W 

(fc) 

7?. 

,3N-\^ 3i," 

■,jN-l) ^ ^0 


positive ground state solution to 
-Am = 5'a,/3,A,/3(^lji^... -f kq;|^ 


in U 


(fe) 


T) 




-Am = ‘^V + kI3^\u\°‘\v\P 

in 

[m > o,M > 0, (m,m) G ^ := X 

ffc) ffc) ffc) 

We can extend every Uj^ ... and v^J... outside ... by 0 and now we 


set 


and 


2«_i 2"-! 


E ••• E e Di'\ng) 

jl=0 jN-1=0 


2 '=-! 2“-! 


„(fc) . 


:=E- E (-1) 

jl=0 jN-1=0 


jlj -hjw-l„,(^) 


v'Z..,N-i^D^o\^e). 


Notice that for any two different choices (ji,^2, • • • , jtv-i) 7^ O'l, J2, • • • , 3 n-i), 
there exists some rotation R G O(R^), the orthogonal transformation, such that 




(fc) 

ilU" ^3N 
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Hence, we have 


Then it follows that weakly solves 

f-Au = + in 

< -Av = + K/3|^|urk|^"^u) in Hg, 

[iu,v) € ^ := £>o’^(He) X DQ'^(He). 


Noting that 2* (s) > 2, after a scaling, let 


«»:= (s„.s.x.Mn5;!-,»-.)) 






2 *(s )-2 


,(fc) 


then {uk,Vk) weakly solves (17.113b . By the construction of Uk and Vk, it is easy to 
see that {uk, Vk) is a sign changing solution and {{uk, Ufc)} are distinct under modulo 
rotations around i/. 

Moveover, we have 


Cfc :=-^ [ [iVitfcp + |Vufep]dx 
^ Jng 


1 


2*{s) 
.1 1 






dx 


) f [|VMfep + |Vufcp]dx 

Jne 


'2 2 *(s)' 


2 «_l 2 '"-l 




ji=0 jAr_i=0 “ii,---,jiv_i 


[|v4".)..,oP + |VuW.,oP]cix 


= (--— 

4 2*{s) 


'^0,... ,0 

,1 1 


(2 “ 


^2 2*{s) 

By Lemma irm Jn^i) as /c —>■ 00 . Recalling that 2*(s) > 2 

again, we obtain that 

C/c 

' + 00 . 


2k{N-l) 


□ 
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Apply the same argument as in the proof of Theorem 17.61 we can obtain the fol¬ 
lowing result for the system defined on 


Theorem 7.7. Assumes € (0,2), N > 4 ,k > 0,a > l,/3 > l,a + /3 = 2*(s). 
Suppose that one of the following conditions holds: 


(oi ) X > fj, and either 1 < jd < 2 or 


/3 = 2 


K > 


( 02 ) X = p, and either min{a, /?} < 2 or 


WIT) 

min{a, /?} = 2, 


K > 


WIT) 


(03) X < p and either 1 < a < 2 or 
Then the problem 


a = 2 


K > 


2*(s) 


'-Am - = Kaj^\u\°-^u\vf in , 

< —Av — pj^\v\'^ = Kflj^\u\°‘\v\^~'^v inR^, (7.115) 

^(M,u)e^:= Dl’^iR^) X Dl’^R^), 


possesses a sequence of sign changing solutions {(uk,Vk)} whose energies . —^ —>• 

2fc(iV —ij 

-\-oo as k ^ 00 , where 


Cfc :=^ / [\yuk\^+ \S/vk\^]dx 
2 Jrw 

2*(s) |a;|" kh 




dx. 


Proof It is a straightforward consequence of Theorems 17.61 and 17.51 We just keep in 
mind that when iV > 4, we have that Sa,p,\,n(Xl-K) = □ 

Remark 7.8. It is clear that this kind arguments used in the proof of Theorem \7.6\ can 
be adapted to other cones with suitable symmetry. 


7.7 Further results on more general domain 1] and on the sharp 
constant 

Remark 7.9. Given a general open domain O (not necessarily a cone), we let Scc,p,\,fi (fl) 
be defined by ifO, 0 and S'a,/ 3 ,A,/i( 0 ) = -l-c». In this subsection, we are con¬ 
cerned with whether Sa,p,\,fi{Id) can be achieved or not and we give some operational 
way to compute the value of Sa,p,x,fji.(^)- 

We note that fl can be written as a union of a sequence of domains, = U^i ^n- 
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Lemma 7.13. Assume D Slj = 0 V z ^ j, then we have 

— inf 

n>l 

Proof. For any n, since C £7, we have 

*^a,/ 3 ,A,/i{^n) ^ A,^(^) for all 71. 


Hence, 


inf Sa 


n>l 


On the other hand, for any e > 0, there exists a pair (w, x;) such that 


(A 


|2*(s) 


a Fl 




(7.116) 


(7.117) 


and 


[ [|VMp + |V7;|2]dx<^„,;3.A.M(H)+e. (7.118) 

JQ 

Set Un = u|j .2 jT’n = f |, since fli fl =0 for all i f j, we have {un,Vn) G 
Dg’^(fin) X andu = Then 


[ [\VUn\‘^ + \VVn\^]dx 

7n„ 


><hc(,/ 3 ,A,/i(H„) I / (A—j—-h/i—j—-1-2 [s)K 


\x\ 


\x\ 


)da;^ 


2 ^ 


>( mf 5'a,/3,A,7i(f^n)) ( / (A 


|2*(s) 


|2*(s) 


\X 

>( mf 5a,;3,A,^(H„))y^^ (A +fi + 2 (s)k; )dx, 




x\ 

2* is) 


\x 

IWnT^nl^ 


^-)dxy 


here we use < 1 and j (A 
It follows that 




- + H- 


lQ.r, 


+ < 1 . 


Jn 

oo 


[|Vu|2-f iVu^dx 


OO p 

= \' I [|Vu„p-f |V7;„p]dx 


> 


ejn 

OO « 

(A 


■, \uJ°‘\Vr 

+ P —rrn-^2 (s)k- 




uc 


uc 


\x\ 


)da 


=( .„t s„,..,„(n„)) f (Ah£^ + ^l£^ + 2 -(.)«hf!^)<i. 

^>1 Jq l^r l^r l^r 


— inf »5’cK^^^A,/i(f^n)- 

n>l 
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Hence, inf„>i Sa,i 3 ,\,tj.{^n) < + £• Therefore, 


inf (7-119) 

n>l 

By (17.1161 1 and (17.119b . we complete the proof. □ 

Next, for r > 0, we set 

:= H n Br, H’’ := fl\Br- (7.120) 

By Remark I tTtI we see that the mapping r i—)► Sa,p,\,fj,Ar) is non increasing and the 
mapping r i—is non decreasing. Hence, we can define 


and 


s^,p,xA^) ■■= i™ s^,P,xA^l- 


Remark 7.10. It is easy to see that p ^{il) and S'^p ^ ^(H) still have the mono¬ 
tonicity property. Precisely, iffli C H 2 , then we have 

slp,xA^i) > sIp^^a^2), s^^p^^a^,) > s^^p^^a^2). 

Theorem 7.8. Assume that s G (0, 2), k > 0, a > 1, /? > 1, a + /? = 2‘*'(s) and H is 
an open domain ofM.^. Let {(m„, Vn)} be a minimizing sequence, i.e., 

I ui. + + 2 ( 5 )'^- = 1 


and 


[ [|Vu„P + \VVn\'^]dx Sa,p,X,t,A) 
JO, 


as n ^ 00 . Then one of the following cases happens: 

(a) There exists some {u,v) G Dq’^(LI) x Dq’^(LI) such that Un ^ u,Vn ^ v 
strongly in T)g’^(f2) and {u,v) f (0, 0) is an extremal function of Sa,p,x,tiA); 

(b) Going to a subsequence if necessary, we set 


p := lim lim 

r—b-0 n—foo 


(A 


+ P- 


X ' 


X' 


, n*: ^ 

+ 2 {s)k -j—- jdx. 


Then 


S%,xA^)ri^ + 5~^,a.;.(11)(1 -v)^ < So.,p,xA^). (7.121) 


70 












Proof. It is easy to see that (u, v) is an extremal function of if and only if 

{u,v) is a ground state solution of 


-Am = ^M + «;q;|^|m|“ in fl, 

-Av = Sa,i 3 ,\,i 3 i^)(^p-J^\v\'^''^‘''>~^v + Kf3j^\u\°‘\v\^-'^v^ in fl, (7.122) 

M > 0, M > 0, (m, m) G ^ := X i7Q’^(0), 


Since {(m„,z;„)} is a minimizing sequence, we have that {(m„,m„)} is a bounded 
{PS)d sequence with d = (2 — 2 «^(s) )Sa,p,\, 0 {^)- Without loss of generality, we 

assume that Un u^Vn v in i7Q’^(0) and Un ^ u,Vn ^ v a.e. in fl. Then it is 
easy to see that (m, v) is a weak solution to (17.1221 1 and 


0 < 


|2*(s) 


.|2*(s) 


+ p- 


+ 2*{s)k 


MT M 


1 ^ 


)da; < 1, 



^ + \Vvf]dx<Sa,p,xA^)- 


Case 1: If (m,m) (0,0), we shall prove that (a) happens. In this case, (u,v) is a 

nontrivial solution or semi-trivial solution of (17.122b . We claim 


|a;|« |a;| 

If not, 0 < /^ < 1. Then 

(f^) 


> 


In {■'*15^ + + 2.(»)Kl±{jli)d. 

/j.j[|VmP -f iVupJda; 


>5'a,/3,A,/i(fl), 

a contradiction. Hence, 

,| 2 *(s) 


.| 2 *(s) 




[\Vuf + \Vv\‘^]dx = Sa.,0,xA^)- 


That is, (m, v) is an extremal function of Sa,i3,x,tJ.{^)- Note that ||u„ — m|| = ||m„|| — 
||m|| -l-o(l), ||m„ - m|| = ||t;„|| - ||m|| -f o(l), we see that m„ u,Vn^ vin L)o’^(H). 
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Case 2: If (u, v) = (0,0), we shall prove that (b) happens. The idea is similar to the 
proof of Lemma l7.7l and Corollarv l7.2l Going to a subsequence if necessary, we set 


and 


Aq := lim lim / [|Vu„p + |Vu„p]c?x 
A°° := lim lim / [|Vu„p + \yvn\'^]dx. 

r—^oo n—>-oo Jqt 


Recalling the Rellich-Kondrachov compact theorem and 2 < 2*(s) < 2* := we 
have {un,Vn) —>■ (0, 0) in x L\^^{Vt) for all 1 < f < 2*. Hence, 


|2*(s) 


-dx = o 


(1). / 

Jn 


|2*(s) 


-dx = o(l) 


Jn FT Jn FI 

for any bounded domain Ct C fl such that 0 ^ (l. Hence, we obtain that 


(7.123) 


L is)K )dx = l-r,. (7.124) 


O'- 


Fr FI 

Similar to the formula (17.521) . we have 

< Aq. 

and similar to the formula (17.531 ). we have 


(7.125) 


(7.126) 


<A“. 

Then by (17.1251 ) and ( 17.1261 ). we have 

< Ao+A“ < (7.127) 


□ 


Theorem 17.81 is a kind of concentration compactness principle, original spirit we 
refer to ll2Tll . Basing on Theorem l7.81 we have the following using result; 

Corollary 7.4. Assume that s G (0, 2), k > 0, a > 1,(3 > 1, a + /3 = 2*{s) and il is 
an open domain o/M^. Then we always have 

(7-128) 

Moreover, if then can be 

achieved. 

Proof. We note that C H, H’’ C H, by the monotonicity property, for any r > 0, 
we have 


Sa,0 ^ Sa,0 ,X,fi{Hr'), Sa,^,X,fi{lf) if Sct,0,X,fj,{H ) 

which deduce (17.128b . Moreover, if 5'a,/3 ^a,m(17) < cnin{S° g ^ S'^g 

then (17.1211 ) will never meet. Hence, only case (a) of Theorerr f7.81 happens. Thus, 

*S'a,/3,A,/i(17) is achieved. □ 
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Furthermore, we have the following result; 

Corollary 7.5. Assume that s £ (0, 2), k > 0, a > 1,(3 > l,a + (3 = 2*(s) and 17 
is an open domain ofM.^. If can not be achieved, then at least one of the 

following holds: 

(i) = Sa,p,\,t,{llr)for'ir > 0 . 

(ii) *S'„,/3,A,/x(^) = 5'a,/3.A./i(17’')/orV r > 0. 

Proof When Sa,p,\,fj,{ld) is not attained, by Corollary 17.41 we have 

So.,p,xA^) = 

Without loss of generality, we assume that ^ ^ ^ a 

Sc.,p,x,f.m = s%,xA^). 

Next, we shall prove that case (i) holds. If not, assume that there exists some tq > 0 
such that Sa 13 X Sa p x /lAro)’ then by the monotonicity property, we have 

5a./3.A.^(17) < S^,p,xAl}ro) < ./3.A.^(^-0) = /3 A^ t;ontradiction. □ 

Remark 7.11. We note that the inverse statement of Corollary 17.51 is not true. For 
example, by Theorem 17.71 when 17 is a cone, Sa,p,x,fiA) attained provided 1 < 
a,l < P,a + (3 = 2*{s),s € (0, 2),k > 0. However, we still have the following 
result. 

Lemma 7.14. Assume that s £ (0, 2),k> 0, a> 1, (3 > l,a + (3 = 2*(s), then 
Sa,p,x,iii^) = Sa,p,x,fj.Ar) = S'a,/3,A,/i(17’')/or any r > 0 iffl is a cone of . 

In particular, 

Sa,p,x,fii'^^) = Sa,p,x,u.{Br) = Sa,p,xA^^\Br) for any T > 0 . 

Further, Sa,p,x,n{13t) = Sa,p,x,fj.{^^) provided that either 17 is a general open domain 
with 0 £ 11 or Q is an exterior domain. 

Furthermore, let Abe a cone ofR^, and 17 = where F is a closed subset of 
A such that 0 ^ F or F is bounded. Then 


Sa,P,X,fi{lf) — Sa,p,X,fi(,A') . 

Proof. We only prove Sa,p,x,A^) ~ Sa,p,x,ii{13r) and the others are similar. By 
the monotonicity property, we see that Sa,p,x,n{13t) < Sa,p,x,fj.Ar)- Next, we shall 
prove the opposite inequality. By Theorem 17.11 Sa,p,x,fj.A) is attained. Now, let 
{u,v) £ Z7g’^(17) X Z7g’^(17) be an extremal function of Sa,p,x,fj.A)- We also let 
XpiA S be a cut-off function such that XpiA = 1 in Be, XpA) = 0 in 
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R^\Bp,\Vxp{x)\ < I and define (jip := Xp(.x)u{x),ij;p := Xp{x)v{x) S C^iVtp). 
It is easy to see that 

/[|V(/)pp + |VV'pP]dx^ [ [\Vu\^ + \Vv\^]dx = 

J Q J Q 


and 


/n ^ |a;h I® 


X 


)da; 


Ax 


=1 


as p —>■ + 00 . Then V e > 0, there exists some po > 0 such that 




“|Tp [Tp [Tp 

Now, consider Mr (a;) := (/>pg(^x),Vr(x) ■.= ipp„{^x) G C'^{Q.r) and 

/Q^[|Vf2r(a;)p + \Vvr{xA]dx 


+ 2*{s)k- 




Sp^a^bi^r') — 


{In. 

/a[|V^PoP + A'^Po\^]dx 


(in ^ 


Hence, 

Especially, take Q = M^, we see that 

<5'a,/3.A,p(R^) = Sa, 0 ,x,p{Br) = Sa, 0 \Br) for any r > 0. 

Hence, when 0 G H, then there exists some r > 0 such that Br C H, then 


Sa,P,X,p{Br) > Sa,p,X,p{^) > Sa,p,X,p{^^) = Sa,p ,X,p{Br) ■ 

If n is an exterior domain, there exists some r > 0 such that (R^\i3r) C H, by the 
monotonicity property again, we have 

Sc,p,xA^ABr) > Sc.,p,xA^) > ^a,/3.A.p(R'^) = S^,p,xA^ABr)- 

Furthermore, A is a cone and H = A\F C A, thenwehave 5'a,/3,A.p(^^) > Sa^p,x,p{A). 
If 0 ^ F, then there exists some r > 0 such that Q,r = Ar, then 


*^a,P,A,p(^) ^ Sct,p,X,pi^r') — Sa^p^x,p{Ar') — Sa^p^x,p{A'). 
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Hence, we have Sa,i 3 ,x,ij,{^) = Sa,i 3 ^\^^{A). If F is bounded, then there exists some 
r > 0 such that H’’ = A’’, then 


Sc.,p,xA^) < Sa,p,xA^l = Sc.,P,X,Ml = s^,p,x,M). 

it follows that Sa,p,xA^) = Sa,p,xAA- n 

To search the results on general domains, we introduce the following marks: 
,4°(H) := {A : A is a cone and there exists some r > 0 such that 


and 


:= {A : A is a cone and there exists some r > 0 such that H’’ C A}. 
Notice that G fl ^ 0, ^ 0. Then we can define 

Slp,xA^) ■■= suv{S^,p,x,M) ■■ ^ e A°m 


and 

■■= sup{5„,^,a.^(A) : A G A°°{n)}. 


Lemma 7.15. 


cO 


m < s^,p,x,,m, s:^,p,xA^) < s^,p,x,,m. 


Proof. We onlyprove < Sl p y^A^). For any e > 0, there exists some 

A G .4°(H) such that 

S%,xA^)-e<So.,p,xAA)- (7.129) 

By the definition of there exists some r > 0 such that C A. Then by the 

monotonicity property, we have 


Sa,P,xAA) < Sa,l3,xA^r) < 5 '°(^)' 


(7.130) 


Bv (17.129b. (17.130b and the arbitrariness of e. we obtain that 5° 

□ 


Remark 7.12. By Corollarv \7.4\ if we can prove that 

So.,p,xA^) < TAAs%^^A^fs’^,0,xA^)}, 

then Sa^p,x,tiA) is attained. Since a lot of properties about cones have been stud¬ 
ied in Section 7 (subsections 7.1-7.6), Lemma \7.15\ supplies a useful way to compute 
min{S'° p ^ ^(17), /i(17)}. Here, we prefer to give some examples. 

Example 1: If Ll is bounded with 0 ^ H, then by S'a,/3,A./j(0) = +oo, we see that 
5'a./3.A.^(^) = 5'“;3 .A.m(^) = 

So.,p,xA^) < ^Hs%,x,f.A)A^,p,xA^)} 


75 








and is attained which can also deduce by Theorem l7.ll 

The following examples are also given by Caldiroli, Paolo and Musina, Roberta 
0, when they study the case of p = 2,b = 0,a G (—1,0). What interesting is that the 
similar results still hold for our case (a slight modification on Example 3). 

Example 2: Assume 0 € is a cusp point, i.e., there exists a unit versor v such that 
V0 £ (0, tt), 3 rg > 0 such that ^ ^e- Notice that 

On the other hand, 

‘5'a,/3, A,/i (^re ) ^ ^ -fOO aS 0 ^ 0. 

Hence, = +oo. 

Example 3: Let O = A x 1 < k < N, where A is an open bounded domain 

of R^. Then there exists some r > 0 such that A C the ball in with radial r. 
Now, we let 


:= {{tx',tx”) eR'^' X R^-'^ 


:t>0,x'G |a;"|Ar-fe > n}, 


then it is easy to see that {A„} are cones such that D A„_|_i,V n and C 

n—1 

iOlxR^~^. Thus. TOea5(n°° , A„) = 0. By (ii)ofLemma l7.1 II lim Sa a x uiAn) = 

+c». Define 0 = B^ x R^“^, then it is easy to see that O C O. Moreover, for any 
n, there exists some r„ > x/r'^ + n? > 0 such that O’’" C where O’’" is de¬ 
fined by (17.1201 1. Indeed, for any x = {xi,X 2 ) £ 0\A„, we have |a;i|fc < r and 
< j, thus \x 2 \N-k < n. Then it follows that |a;|Ar < x/r'^ + n^. Hence, 
O’’" C O’’” C An- Then by the monotonicity property we have Sa,i 3 ,x,fj.{^''^) > 
-S'a,/3,A,/i(0’’”) > Sa,p,x,tJ.{An) -)■ +00 as n -)> + 00 . Hence = 00 . 

Lemma 7.16. Assume that s £ (0,2),/t > 0,a > l,/3 > l,a + /3 = 2*(s), and 
O = A X 1 < fc < W, where A is an open bounded domain o/R^ with 0 ^ A. 

Then Sa^/ 3 ^x,fj.(S^) is attained. 

Proof. By Example 3, we see that 5')^^ ;^_^(0) = -boo. ByO ^ A, we have 5°^^ ;^_^(0) = 
-boo. Then by Corollary 17.41 we obtain the conclusion. □ 

Based on the result of Lemma l7. 141 and the maximum principle, we can obtain the 
following interesting results. 

Corollary 7.6. Assume that N > 3, s£ (0, 2 ),k> 0, a> l,/3> l,a-b/3 = 2*(s), 
and let fl be a general open domain ofM.^. 

(i) Iff) £ O, then S'a„a,A,/i(0) is not attained unless O = R^; 
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(ii) IfQ. is an exterior domain, then is not attained unless ; 

(Hi) IfVL = A\JU, where U is an open bounded set with 0 ^ U and A is a cone of 
then 

So.,p,xAn) < = s°,^,xA^) = ^^/^,xA^) 

and Sa,p,xA^) attained. 

Proof. For the case of (i) and (ii), by Lemma l7.141 we see that 

So.,P,xA^) = So.,I3,xA^"')- 

Then by the maximum principle, it is easy to see that Sa,p,x,ti(,^) is not attained un¬ 
less il = . For the case of (Hi), since U is bounded and 0 ^ U, there exists 

ri,r 2 > 0 such that = Ar^,n^^ = Hence = 

Sa,l3,xAA- If S'a,/3,A,M(^) < “^{'5'°(f^)> ^orollary 

17.41 Sa,i3,x,tJ.{^) is attained and the proof is completed. If not, by Corollarv l7.4l again. 
5'a,/3,A,/j(fI) = ‘^^/3,A,/j(fI)}’ Hsnce, Sa,/3,X,fj.{^) = Sa,/3,X,fJ.A)- 

By maximum principle again, Sa,p,x,AA attained, a contradiction with Theo¬ 
rem ITT] □ 

Theorem 7.9. Assume that TV > 3, s G (0, 2), k > 0, a > 1, /3 > 1, a -I- /? = 2*(s) 
and H C R^ is an open bounded domain. IfO G H, then Sa,p,x,ii{^) H not attained. 

Proof. Since H is bounded, there exists some r > 0 such that H C Br(0). When 
0 G H, by Lemma 17.141 we have 

<S'a,/3,A,/i(fI) = Sa,0,X,/i(Br(O)) = Sa,/3,X,f.t(^^) ■ 

By way of negation, assume that Sa,/3,x,fj.i^) i® attained and let (uju) G dAA) X 
Dq’'^A) bs an extremal function. We may assume that u > 0,u > 0. By extend¬ 
ing u and V outside H by 0, then we see that (u, v) is also an extremal function of 
Sa,i3,x,ti{Br{0). Hence, {u, v) f- (0,0) is a nonnegative weak solution to the follow¬ 
ing problem; 

f-Au = 5'a,/3,a,b(-Br(0))fA|^|upTs)-2y _|_ |y |a-2y |^|/3^ jjj Br(Q), 

I -Av = Sa,p,a,biBriO))yp.j^\vA^'''>~'^V + k/ 5 |u|“ |u in Br{0), 

[u > 0,u > 0, (u,z;) G ^ := DA{Br{0)) x DA{Br{0)). 

(7.131) 

On the other hand, since Br{0) is a star-shaped domain, by Proposition 17 .3 1 (see the 
formula ( 17.28b ). problem (l7.131b has no nontrivial solution even semi-trivial solution, a 
contradiction. Hence, we know that 5'a,/3,A./j(fI) is not attained. □ 

Corollary 7.7. Assume that N > 3, sG (0, 2 ),k> 0, a> l,/3> l,a + (3 = 2*(s), 
if there exist some ri, r 2 >0 and 6 G (0, tt] such that 

Ae n Br, (0)) C H C (He n Br, (0)), 

then S'a,^,A,/i(fI) = Sa,i 3 ,x,iJ,A‘e) and is not attained. 

Proof. The proof is similar to that of Theorem l7.9l we omit the details. □ 
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8 The case of Si S 2 G (0, 2) 


In this section, we study the case of si ^ S 2 G (0,2). By constructing a new ap¬ 
proximation, the existence of positive ground state solution to the system (II.lb will be 
obtained, including the regularity and decay estimation. 

Assume 17 is cone. Define 


A 

where [/ is a ground state solution to the following problem: 

f — Am = /isj (17) — in 17, 


I M = 0 on 917. 


lap 


Set 


where 


On := 


771,0 := in,f IImIP, 772,0 := inf \\uf, 

v£=.q liGBo 

:= {m e i:>o’^(17) : = l|, 






The corresponding energy functional of the problem (II. lb is defined as 

1 

2^ 


T>o(m,m) = ^a{u,v) - - Kc{u,v) 


for all (m, v) G where 

a{u,v) := ||(m,m)|||, 

6(m, v) := A + 77 

^c(m,m) := J^j^\u\°‘\vfdx. 

Here comes our main result in this section: 


|d| 


2*(si) 


dx. 


( 8 . 1 ) 

( 8 . 2 ) 

(8.3) 

(8.4) 

(8.5) 

( 8 . 6 ) 


Theorem 8.1. Assume that Si,S 2 G (0,2),A,/i G (0 ,+oo),k > 0,a > l,/3 > 
1, a + /3 = 2 *(s2)- Suppose that one of the following holds: 

f/3 = 2 

(oi) A > 77 and either 1 < /? < 2 or < 

> 2 ^ 


( 02 ) A 


77 and either min{a, /?} < 2 or 


min{a, /?} = 2 , 

\ m,o _ m,o 
^ ^ 2*(s2) “ 2*(s2) 


( 03 ) A < 77 and either 1 < a < 2 or 


a = 


2 


K > 


m,o 

2*(s2) 
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Then problem (fTTT i possesses a positive ground state solution (uq, Vq) such that 



Moreover, we have the following regularity and decay propositions: 


(bi) ifO < maxjsi, S2} < then uq, vq € C^(ri); 

(62) !/max{si,S2} = thenuo,vo G {Vl) for allO < 7 < 1; 


(63) !/max{si,S2} > thenuo,vo G C^’~^ {M.)forallQ < 7 < 

When r2 is a cone with 0 G dV, (e.g., ), then there exists a constant C such that 



|Mo(a;)|, |'yo(a:)l < <^(1 + k| |VMo(a:)|, |Vi;o(a:)| < C|a;| 

When ft = R^, 

|■uo(a;)|, |z;o(a;)| < C {1 + |a;|“^), |Vuo(a;)|, |Vz;o(a:)| < C\x\~^~'^ 


In particular, ifVl = R+, then (uo(x), t’o(^)) axially symmetric with respect to the 
XN-axis, i.e., 

(Mo(a:),i'o(a;)) = {uo{x ,xn):Vo{x',xn)) = (Mo(|a:'|,a;Ar),?;o(|a:'|,a;iv))- 

Remark 8.1. The regularity, symmetry results and the decay estimation we have es¬ 
tablished in Section 3 of the present paper. Therefore, in the current section we only 
need to focus on the existence of the positive ground state solution. 


8.1 Approximation 

When Si f si, the nonlinearities are not homogeneous any more which make the 
problem much tough. Here we have to choose a different approximation to the original 
problem in the same domain, i.e., we consider the following problem: 


—Au — A = Kaaeix)\u\°‘ in H, 

—Av — = KPae{x)\u\‘^\v\^~'^v in H, (8.7) 

K > 0, (u, v)G^:= X 


where 



( 8 . 8 ) 


Under some proper assumptions on a, /3, A, p and /t > 0, we shall prove the existence 
of the positive ground state solution (ue, vf) to (18.7b with a well-dominated energy (see 
Theorem l8.2l below). Finally, we can approach an existence result of ( II.lb . 
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The corresponding energy functional of problem (18.7b is defined as 

= ^a{u,v) - - K,Ce{u,v) 

for all (u, v) G 3l, where a{u, v) and b{u, v) are defined in (18.6b and 

c^{u,v) := [ ae{x)\u\°‘\vfdx, 

Jo, 

which is decreasing by e. Consider the corresponding Nehari manifold 
Me ■■= {{u,v) G ^\(0,0) : Je{u,v) = 0} 


(8.9) 


( 8 . 10 ) 


where 

Je{u, v) v), (u, v)) = a(u, v) — b(u, v) — K(a + /3)ce(u, v). (8.11) 

Lemma 8.1. Assume si,S 2 G (0,2),A,p G ( 0 ,+oo),k > 0,a > 1,/3 > 1 and 
a + f] = 2*(s2). Let e G [0,S2)> then for any {u,v) G ^\{(0, 0)}, there exists a 
unique t = t(^e,u,v) > 0 such that {tu, tv) G Me- Moreover, Me is closed and bounded 
away from (0, 0). Further, t = t(^e,u,v) is increasing by e. 

Proof The existence and uniqueness of f = t(^e,u,v) that Me is closed and bounded 
away from 0, we refer to Lemma I tTI Now, we prove that t = t(^e:,u,v) is increasing by 
£. Assume that 0 < ei < £2 < S 2 , then we see that there exists a unique ti and t 2 such 
that 

Je^{tiU,tiv) = Je.^{t2U,t2v) = 0 . ( 8 . 12 ) 

Recalling that Ce{u,v) is decreasing by e, we see that Je{u,v) is increasing by e. 
Hence, 

Je2{tiU,tiv) > Je^{tiU,tiv) = 0. (8.13) 

If Je 2 {tiu, tiv) = 0, by the uniqueness, we obtain that t 2 = t\. If Jej {tiu, t\v) > 0, 
noting that Je^itUjtv) — >■ —oo as f — >■ +oo, there exists some t* > ti such that 
Je 2 {t*u,ttv) = 0. Then by the uniqueness again, we see that t 2 = > ti. Hence, 
we always have t 2 > fi and we note that t 2 > ti when uv ^ 0. □ 


Define 


Ce := inf 

{u,v)£Me 




4 : = 


inf 


vImFTnP 


(8.14) 


We have the following results: 

Lemma 8.2. 5e is increasing by e £ [0,S2), i.e., Jq < Se^ < Se^ provided 0 < £i < 
£2 < S2- 


Proof For any {u,v) f (0,0), set f Lemma 

18.11 there exists 0 < ti < t 2 such that {tif, tifi) G A4i and (f 2 </>, ^2^’) £ ■i^e 2 - Hence, 
we obtain that 6e is increasing by £ G [0, S2). □ 
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Remark 8.2. Set s 

we have 




:= inax{si, S 2 }. it is easy to prove that for any {u, v) € Me, 

{u,v) > (^ - -r) (l|w|P + IkiP) (8-15) 

\ Z Z \Smax ) ' 


and it follows that 



2*{i 



(8.16) 


Lemma 8.3. is increasing by e in [0, 32 )- 


Proof. Let ip) ^ (0, 0) be fixed. By Lemma ISTl for any e € [0, S 2 ), there exists a 
unique te > 0 such that te{(p, ip) € Me- In fact, te is implicitly defined by the equation 


a{(p,ip) - b{(p,ip)tT^'‘^'' - 2 *{s 2 )KCei(p,ip)tf^'‘^'> ^=0. (8.17) 


It then follows that 


1 


rl 


L2 2*(a2) 


a{(pM)[t{e)] 


^e{t{e)(p,t{e)ip) 

1 1 


L2*(a2) 2*(si) 


Case 1: S 2 > si. For this case, we see that 


2*(s2) 2 *(si) 

a{(p, Ip) > 0, b{(p, Ip) > 0 and Lemma lOl we obtain that 

$£(t(e)(^,f(e)'0) is increasing by e in [ 0 , 52 ). 


(8.18) 

^ >0. Noting that 


(8.19) 


Hence, we get that is increasing by e in [0, S 2 ). 

Case 2: S 2 < si. By the Implicit Function Theorem, we see that t{e) G C'^(]R) and 
^f(e) > 0 by Lemma lsn Hence, 

f<S>e{tie)(p,t{e)ip) 

=2[i - ^^Hf,me)t'{s) + 2*(si)[^ - ^m,ip)mf^^^^-^t\e) 

“lit'' - 


f{e) 


[ 1 - 


t{e) L 2 *(s 2 ) 
, r2*(si) 


\a{f,ip)[t{e)]'^ 


'2*(s2) 


- 1] [a(</>,^) [*(£)] ^ - ‘^*{s2)KCe{(p,ip)[t{e)]^*^"^'i] 


t'{e) r 2*(si)-2 
t{e) I 2 *{s 2 ) 
> 0 . 


a{cP,iP)[t{e)r + [ 2 *( s 2 ) - 2*{s,)]KCe{cP,iP)[t{e)f^^^^ 


( 8 . 20 ) 


Hence, we also obtain the conclusion of ( 18.191 ) for the case of S 2 < si and the proof is 
completed. □ 
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8.2 Estimation on the least energy of the approximation 

Recall U\ := jj^ where 17 is a ground state solution of the following 

problem: 


— Am = /ijjj (n)2^-rrTir— in 17, 


M = 0 on 517. 


Define the function 

'I'At'Mj = 


(m) = 


dx — 




A 


J2*(si) 


2*(si)ya 


-dx. 


( 8 . 21 ) 


( 8 . 22 ) 


Then 


mA = «'a(CAa) = [2 - ^ 


(si) 


](/^si(17))' 


•(«i) 

>T7^ 




(8.23) 


is the least energy. 

Remark 8.3. Evidently, for any e £ [0, S 2 ), we have that < mx and Cg 
Hence, 



](m^i (17)) ' (max{A,Ai})’ 


2*(si) 


2*(si)-2 


< 

(8.24) 


Define 

7i,e := inf ||u||2 


□ 

(8.25) 


where 

E,-.= {v £ : [ a,{x)\Uxf^^^'>-M'^dx = l}. (8.26) 

Jn 

Since ae(a:) is decreasing by e, it is easy to see that pi ^ is increasing by e. 

Lemma 8.4. Assume si,S 2 G (0,2),A,/r G ( 0 ,+oo),k > 0,a > l,/3 > 1 and 
a + /3 = 2 *(s2). Let e £ [0, 52 )- 

(1) If (3 < 2, then < mx. 

(2) If (3 > 2, then (Ux, 0) is a local minimum point o/$e in Mg. 

(3) If (3 = 2 and k > Cg < mx. 


(4) If (3 = 2 and 0 < k < 2 * { ’ 32 ) ’ f*) local minimum point of in Me. 

Proof. The proofs are similar to those in Section 6.2. □ 

Lemma 8.5. 771 is continuous with respect to e £ [0, 52 )- 
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Proof. For any £q G [0, S 2 ), we shall prove that ^ is continuous at e = £q. Apply 
the argument of Lemma |63] there exists some 0 < uq G such that 

Ikof = m.eo and / = 1. (8.27) 

JQ 

Take a sequence {£n} C [0, S 2 ) with e„ J, eo as n —>■ + 00 . Recall that is increasing 
by £, then lim ? 7 i,e„ exists and satisfies 

n—>-+oo 

lim (8.28) 

n—>-+oo 

On the other hand, since (x) ao{x) a.e. in O, recalling the decay property of U\ 
(see lfT3l Theorem 1 . 2 ], |[T5] Lemma 2 . 1 ], ||T9] Lemma 2.6]), it is easy to prove that 


lim / ae„(a;)|i7Ap*^®^^ '^v'^dx = / 050 ( 2 ;)|La= 1 . 

Jn 

Hence, 

IkolP 

n->.+oo 

Then by the definition of pi 5 , we see that 


lim pi,e„ 

n—>-+00 


< lim 


Ikoll^ 


->■+00 J^a^^{x)\U\\'^'‘^‘‘^^~'^VQdx 


By (18.28b and (18.31b . we obtain that pi ^ is right-continuous. 


m,eo- 


(8.29) 

(8.30) 


(8.31) 


Secondly, we take a sequence {£n} C [0, S 2 ) such that e„ f £0 as n —+ 00 . By 
Lemma Lemma |63] again, we may assume that {u„} C 79g’^(H) such that 

Iknf = 7 i,£„ and / (a:)|C/Ap*^'*"^"^u^da; = 1. (8.32) 

Jo. 

Up to a subsequence, we may assume that Vn vq in Z)g’^(H) and Vn vq a.e. in H. 
Similarly, we can prove that 


/ a^g^xfllx]"^ ^v^dx = lim [ aE„(x)|LA|^ = 1. (8.33) 

Ja n-s.+oo 

(8.34) 

n—>--|-oo 

Ikof 


In 

It follows that 


Therefore, 


< liminf ||u„|| = lim 

n—f-\-co n—b-+oo 


m,eo < 


< lim pi,E„. 


(8.35) 


J^aeo{x)\Ux\'^’('^^')-'^V^dx n->-+oo 

On the other hand, by the monotonicity, we can obtain that reverse inequality. Hence, 

~ n 4 ’?oo 

i.e., ryi 5 is left-continuous. The proof is completed. □ 


83 










Similarly, we define 


(8.36) 


m,e ■= inf ||m|| 

iiG0e 


where 


e,:= ^a,(x)|t/^p*(^^)-2|u|2da; = l}. (8.37) 

We also have that 772 ,£ is increasing by e € [0, S 2 ) and continuous. Furthermore, we 
can propose the following results without proof. 

Lemma 8.6. Assume si,S 2 G (0,2),A, p G (0, +oo),k > 0,a > 1,/3 > 1 and 
a + P = 2*(s2). Let e G [0, 52 )- 

(1) If a < 2, then Cg < m^. 

(2) If a > 2, then (0, U^) is a local minimum point o/$£ in Me- 

(3) If a = 2 ,k > 2fsl) ’ 

(4) If a = 2, 0 < K < 2 *l’s 2 ) ’ local minimum point of^e in Me- 

Now we can obtain the following estimation on Ce'- 

Lemma 8.7. Assume si,S 2 G (0,2),A,p G (0, +oo),k > 0,a > 1,/3 > 1 and 
a + /3 = 2 *(s 2 )- Let e G [0, S 2 ), then we have 


Ce < min{mA,TO^} = 


2*(ai) 


_2 2*(si)J 

if one of the following holds: 

(a) X > p and either 1 < /? < 2 or 


{psi (fi)) ( max{A, /i}) 


2^ 


/3 = 2 


K > 


2*(s2) 


(6) X = p and either min{a, /?} < 2 or 

"a = 2 


minjo;, /?} = 2, 


n ^ _ m,e 

^ 2*(s2) 2*(s2) 


(c) X < p and either 1 < a < 2 or ■ 


K > 


2*(s2) 


Proof- It is a direct conclusion following by Lemma [8^ and Lemma lS^l 


□ 
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8.3 Positive ground state to the approximation problem (18.71) 

In this subsection, we assume that e G (0, S 2 ) is fixed. Then we can obtain the follow¬ 
ing result. 

Theorem 8.2. Assume si,S 2 £ (0,2),A,^ £ (0,-|-oo),k > 0, a > 1,/3 > 1 and 
a+P = 2*(s2). Then problem (I8.7l i possesses a positive 2 roundstate solution (6^. ib^) 
provided further one of the following conditions holds: 


(1) \ > p, and either 1 < j3 < 2 or 


/3 = 2 


K > 


(2) X = p, and either min{a, /?} < 2 or 


2*(s) 

min{a, /?} = 2, 


2*(s) 2-{s) 


(3) X < fj, and either 1 < a < 2 or 


a = 2 


K > 


V2.e 

2-(s) 


Proposition 8.1. Assume that e £ (0, S 2 ) and {(u„, Wn)} is a bounded {PS)c sequence 
0 /$£. Up to a subsequence, we assume that ^ {4’,'f) weakly in St. Set 

Un ■= Un — (j), Vn ■= Vn ~ 'f, then we have that 

'^'xiun) 0 and ^ 0 in (8.38) 

where T'a is defined in (18.221 1. 

Proof Under the assumptions, we see that 

(<i>'(zi„,t;„),(fi,0))=o(l)||/i|l (8.39) 

Since (m„, u„) ^ {f, ip), it is easy to see that ip) = 0. Then we have 

{^',{^,iP),ih,0))=0. (8.40) 

By Lemma POl and Holder inequality, it is easy to see that 

f ae{x)\un\°‘~'^Un\vnfhdx - f as{x)\(p\°‘~'^(p\ipf hdx = o{l)\\h\\. (8.41) 

Jq Jq 

It follows from (18.39b . (18.401) and (18.41b that 


[ S7{un-<p)S7hdx-X [ (- 

Jq Jq ^ 


By IfT^ Lemma 3.3] or [S] Lemma 3.2], we see that 


-'jhdx = o(l)||/i||. 

(8.42) 






mH-^{n). (8.43) 


Ispi 

Hence, by (18.42b and (18.43b . we obtain that 

'ii'xiun) ^ 0 in H-\n). (8.44) 

Apply the similar arguments, we can prove that (fin) 0 in □ 
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Corollary 8.1. Under the assumptions of ProDosition \KT\ and furthermore we assume 
that 


c < min{mA,m^} = 


1 


1 


(/isj (f2)) 


2*(«l) 

2’(si)-2 


(max{A, /i}) 


2*(n)-2 


_2 2*(si)_ 

Then up to a subsequence, {un,Vn) (</>, "0) strongly in fSi and {(j), ip) satisfies 
((/), ^) = c and ifi), Ip) = 0 in S!*. 


Proof. We prove it by way of negation. Assume that (m„, Vn) {(p, ip), then at least 
one of the following holds: 

(i) Unjf (pin £)q’^(0); 

{a) Vnyf Ip in L)o’^(O). 

Without loss of generality, we assume (i). By Proposition IS.ll we see that —>■ 0 


in H ^(O). Since Un = Un — (p -ft 0 in Dg’^(n), it is easy to see that 

liminf 'I'A(Mn) > mx- (8.45) 

n—>-+oo 

On the other hand, by the Brezis-Lieb type lemma (see m Lemma 3.3]), we have 

^eiUn,Vn) = ^e{Un,Vn) + + o(l). (8.46) 

By Lemma lTSl again. we see that 

4>e(u„, Vn) = ^x{Un) + fj.{Vn) + o(l). (8.47) 

Since 0 in H~^{n), it is easy to prove that liminf 'l'u(f'n) > 0. We also 

note that ^e{(p, ’P) > 0. Then by (18.461) . (18.471) and (18.451) . we have 

c= lim ^e{un,Vn)> Hm '^x{un)>mx, (8.48) 

n—>-+cxD n—>-+cxD 

a contradiction. □ 


Proof of Theorem 18.21 Let {{un,Vn)} C A4 be a minimizing sequence. Then it is 
easy to see that 

^e{un,Vn) “)• and -)■ 0 in 

It is standard to prove that (m„, Vn) is bounded in ^ and is also a {PS)c^ sequence of 
$£. By Lemma [HIT] we have 


< min{mA,m^} = ^ 


(■si) 


KtsA^)) 


2^ 


(■il) 




2*(«l)-2 


(8.49) 
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Hence, by Corollary 18.11 there exists some {(j), ifj) G ^ and up to a subsequence, 
{un, Vn) —>■ (</), 't/j) Strongly in Moreover, we have $e(^, '0) = Ce and ip) = 
0. Thus, (0, Ip) is a minimizer of Cg. It is easy to see that (|0|, \ip\) is also a minimizer. 
Hence, without loss of generality, we may assume that 0 > 0,0 > 0 and it follows 
that (0, Ip) is a nonnegative solution of ( I8.71 i. Recalling ( 18.491 ), it is easy to see that 0 ^ 
0,0 ^ 0. Finally, by the strong maximum principle, we can obtain that 0 > 0,0 > 0. 
That is, we obtain that (0, ip) is a positive ground state solution of (18.71 ). □ 


8.4 Geometric structure of positive ground state to (18.71) 

Now, let us dehne the mountain pass value 


Ce := inf max ^^{^{t)), 
■yer^ tG[o,i] 


(8.50) 


where Te := {7(f) G 1],^) : 7 ( 0 ) = (0,0), $e( 7 (l)) < 0}. We have the 
following result. 

Theorem 8.3. Assume si,S 2 G (0,2),A,p G (0,+oo),k > 0, a > 1,0 > 1 and 
a + 0 = 2* ( 52 )- Tef e G (0, S 2 ) cind one of the following hold: 

[0 = 2 

(0 \> jjL and either 1 < 0 < 2 or < 

> m,e 


(a) X 


jjL and either min{a, 0} < 2 or 


min{a, 0} = 2, 
> Vl,e = V2 ,e 


{in) \ < gL and either 1 < a < 2 or < 

V2,e 

Then Cg = and any positive ground state solution of system (ISJt is a mountain pass 
solution. 


Proof It is easy to check that satishes the mountain pass geometric structure. Re¬ 
calling the existence result of Theorem 18.21 let (0,0) be a positive ground state so¬ 
lution of (18.7b . Dehne 70 (f) := tT{(p,ip) for some T > 0 large enough such that 
4 >e(T0, Tip) < 0. Then it is easy to see that 70 G Tg. By Lemma lOl we have 

$e(0,0) = max$E(f0,f0). (8.51) 

Hence, 

Ce < = '^e{(p,1p) = Ce- (8.52) 

Under the assumptions, it is standard to prove that Cg is also a critical value and there 
exists a solution {(p,ip) such that ^g{(p,ip) = Cg and $(.(0,0) = 0 in Then we 
see that (0, ip) G Afg. Hence, by the dehnition of Cg we see that 

Cg := inf $e(it,?;) < $e(0,0) = Cg. (8.53) 

{u,v)eMs 
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By doa and (l83^ . we obtain that = c^. For any positive ground state solution, 
by the arguments as above, we have the mountain path 70 S F^ and thus, the positive 
ground state is indeed a mountain pass solution. □ 

Remark 8.4. 

(i) Recalling that for e € [0, S 2 )> both and r] 2 ,e increasing by e and contin¬ 
uous with respect to e. When k > S {1, 2}, then by the continuity, we 

see that k > 2 *( 32 ) ^ small enough. 

(ii) Note that the proof of for e € (0, S 2 ) depends heavily on the existence 

of the ground state solution. When e = 0, the existence of ground state solution 
is still unknown. However, we will prove that the result cq = Cq is also satisfied 
(see Corollarv \8.2\ below). 

Lemma 8.8. Cg > cq and lim = cq 

e—>0+ 

Proof. By the monotonicity of ae{x), it is easy to see that > Cq. Hence, 

lim Ce > Cq. (8.54) 

£—>■0 + 

Next, we only need to prove the inverse inequality. For any (5 > 0, there exists 70 G Fq 
such that 

max 4>o(7o(f)) < Co + <^- (8.55) 

te[o,i] 

Denote 7 o(l) := {f, fi), since 70 G Fq, we have tp) < 0. 

Case 1: If = 0, it is easy to see that $£((/), •0) = < 0. Hence, 

7 o G Fg for all e G [0, S 2 ) for this case. 

Case 2:If ^ 0, then by the Lebesgue’s dominated convergence theorem, we 

have 

lim [ ag{x)\(l)\°'\'ip\^dx = [ ao(a;)|(/>|“|^/’|^fia;. (8.56) 

£->■0+ Jn Jn 

Hence, we have $g((/>,'!/;) < 0 when e is small enough. Thus, we also obtain that 
7 o G Fg when e is small enough. Now, we take an arbitrary sequence e„ 4- 0 
n —>■ + 00 . Choose tn G [0,1] such that 

^e„( 7 o(fn))= max $g„( 70 (f)). (8.57) 

46 [ 0 , 1 ] 

Up to a subsequence, we assume that —?► f* G [0,1] and denote that 

loitn) ■= 7o(f*) := iu*,v*). (8.58) 

Since 70 G C([0,1], ^), we obtain that (u„, u„) —>• {u*,v*) and it follows that 

u„) = $g„ {u*,v*) + 0 ( 1 ). (8.59) 

By the Lebesgue’s dominated convergence theorem again, we have 

$g„(u*,u*) = $o(u*,«*) + o(l). ( 8 . 60 ) 
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Hence, by ( 18.59b and (18.60b . we have = $o(«*,!'*) + o(l). Then 

Ce„ <^'e„(7o(in)) = ^eAUn.Vn) 

=^o{u*,v*) + o(l) = $o( 7 o(i*)) + o(l) 

< niax $o( 7 o(i)) + o(l) = V') + o(l) 

tG[0.1] 

<co + (5 + o(l). (8.61) 

Let n —+oo, we obtain that lim Ce^ < cq + S. Hence, lim£_>Q+ < cq. Insert 

_ n—>-+oo 

( 18.54b . we complete the proof. □ 

Corollary 8.2. cq = cq and lim = cq. 

£—>■ 0 + 

Proof. For any (it, v) ^ (0, 0), define 7 (f) = f(it, v), then we see that 7 S Fq. Hence, 
it is easy to see that cq < cq. On the other hand, by Theorem 18. 3 1 and Lemma ISTSl 
we have cq = lim£_,,o+ = lim£_,,g+ c^. By Lemma 1831 we have lim£_,,Q+ > cg. 

Hence, we obtain that cg = Cg and lim = Cg. □ 

£—>■0 + 


8.5 The existence of the positive ground state to the original system 

Take {£n} C (0, S 2 ) such that e„ | 0 as n —>■ + 00 . By Theorem 18.21 the system ( 18.7b 
possesses a positive ground state solution (ii„, Un). By Remark [831 we have 

C£„ = $£„(M,t^) > (8.62) 

\ Z Z \Smax ) ' 

By Corollary 18.21 we have Ce„ cg. Hence, {{un,Vn)} is bounded in Up to 
a subsequence, we may assume that (ii„,i;„) ^ (ug.ug) weakly in ^ and it„ —>■ 
uo,Vn Vo a.e. in H. We shall establish the following results which are useful to 
prove our main theorem. 

Lemma 8.9. (ug, vg) satisfies ^q{uo,vq) = 0 in 
Proof. We claim that for any € ZlQ’^(f2), we have 

lim [ as„{x)\un\°‘~'^Un\vnf(j)dx= [ ag(a;)|Mo|““^Mg|i;g|^(/)da;. (8.63) 

Without loss of generality, we may also assume that (j> > 0. If not, we view = 
(j)+ — (j)-, and we discuss (j)+ and respectively. 

Firstly, by Fatou’s Lemma, we have 


ao(x)|itg|“ ^Mojugl^^dx < liminf / ag^{x)\un\°‘ '^Unlvnl^fidx. (8.64) 


n—>-+oo 


On the other hand, since (x) < ao(x), we have 

/ ae„(x)lunr~^u„lv„l^fidx < / ao(x)|u„|““^it„|i;„|^(/)da;. 
Jq Jq 


(8.65) 
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Further, since (it„, u„) ^ {uq,vo) in it is easy to see that 

2 *(« 2 ) 


|u„|“ '^Un\vnf ^ \uo\°‘ in {n,ao{x)dx), 

then we have 

lim [ao{x)\un\°‘~‘^Un\vnf(t)dx= [ ao{x)\uo\°‘~‘^uo\vof4>dx. (8.66) 


By (ESSll and (ESUl, we have 


limsup / ae„(a;)|M„|“ < / ao(a;)|uor UQ\vo\d(j)dx. (8.67) 

n—)-+oo JQ, Jfl 


Hence, from ( 18.64b and ( 18.67b . we prove ( 18.63b . 

Similarly, we can prove that for any ip € £)q’^( 0), we have 


lim [ ae„{x)\un\°'\vnf '^Vn’ipdx = [ ao{x)\uo\°'\vof ‘^VQipdx. (8.68) 

„_j.+oo 


IQ JQ 

Recalling that (u„, Vn) are critical point of <l)e^, for any ((/), ifj) G we have 

('/'>'*/'))= 0- (8-69) 

Then by (18.63b . (18.68b and (m„, Vn) ^ (uq, vq) weakly in we obtain that 

(<^'o(uo,vo),(iP,iP)} = 0. (8.70) 

Hence, <i)g(uo, uq) = 0 in □ 

Lemma 8.10. If{uo,vo) 7 ^ (0,0), then $o(wo,'fo) = cq > 0. 

Proof. Since (un, u„) is a positive ground state solution of (Pe„), it is easy to prove 
that 


Ce„ = ^e„(Mn,Wn) = 


1 1 


L2 2*(si)J“^ " 

By Lemma [ 8 ( 9 ] we also have 


r2*(52) 1 

^ 1 (Wti,,'U yj). (8.71) 


^o(Mo,'yo) = 


rl 


6 (mo,uo) 


-2 2 *(si) 

Noting that by Fatou’s Lemma, we have 


2 *(s 2 ) 


- 1 


and 


b{iio,vo) < liminf 

n—>-+oo 


co{uo,vo) < liminf C£„(u„,t;„). 

n—>-+cxD 


kcq{uo,Vo). (8.72) 


(8.73) 


(8.74) 
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Hence, $o(wo, Wo) < 1™ c, 

_ n—>-+oo 

18.21 we have lim c, 

n—>-+oo 


and $o(wo, vg) > 0 follows by (18.72b . By Corollary 
= Co, hence 


^o(wo,wo) < cq. (8.75) 

On the other hand, when (uo,vo) ^ (0,0), it is trivial that <l)o(wo,wo) > cq > 0 
by Lemma [8^ and the definition of cq. Hence, we obtain that <I)o(uo,wo) = cq if 
{uo,vo) ^ {0,0). □ 

Lemma 8.11. 


3>o(wo,wo) <min{mA,TO^} 
'1 1 


2 *(«l) 


(msi(^^)) ' (max{A, Ai}) 


■ 2 *(n )-2 


.2 2*(si)J 

Proof. It is a direct conclusion by Lemma lS/Tl and Lemma l8. lOl □ 

Corollary 8.3. If{ug, vf) (0,0), then {uq, vq) is a positive ground solution of (II.lb . 


Proof. Since (u„, w„) are positive and —>■ ug, w„ —>■ Vg a.e. in H. We see that ug > 
0, Wo > 0. If Wo = 0, then we see that 4''^(ito) = 0 and ug 0. Hence, <i>o('Uo, wq) = 
'I'a('Uo) > a contradiction to Lemma lS.llI Similarly, if ug = 0,vg 0, we see 

that $o(wo,wo) > m^, also a contradiction to Lemma lS.llI Hence, uo 0,wo ^ 0 
and $o(wo, wo) = cq by Lemma IS. 101 That is, (ug, vg) is a nontrivial and nonnegative 
ground state solution of (foi l. Finally, by the strong maximum principle, we can prove 
that (wo, Wo) is a positive solution. □ 


Lemma 8.12. Assume that lim inf / ae„|u„|“|w„|^ = 0, then T'(^(u„) —>■ O,'?'(w„) —>■ 

n—>+00 /q ^ 

Oinff-^(fl). 

Proof. Under the assumptions, we claim that up to a subsequence. 


/ |a£„(x)|u„|“ ^w„|w„|^/i|da; = o(l)||/i||. 

in 

For any h G Zlg’^(r2), since ae„{x) < ag{x), we have 
f ae.^\hf [ ao|/i|^ 

j n J Cl 


(8.76) 


Then by the Hardy-Sobolev inequality, we obtain that there exists some C > 0 inde¬ 
pendent of n such that 


( / aejhf^'‘^'>dx'^ ' < C\\h\\. 

J Q 


(8.77) 
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Noting that + 2 »(s 2 )a 2 *{s 2 ) ~ Holder inequality and (18.771) . we have 




Un Ur,m 


il^hlda: 


< 


aen\Un\ \Vn 


fdxy 


J ae„{x)\vnf ^ J ae„|h|^ 

= o(l)||/i||, (8.78) 

which means that (18.76b is proved. Since {un, Un) is a positive ground state solution of 
the system (18.7b . we have 


(h, 0)) = 0. 


That is, 


In 


VunV/i —A— - —:—r-- —h — Ka I aF:„{x)\ 


X ■ 


Uni Un\lln 


In 


fh 


for all n and h G Then by (18.76b and (18.80b . we obtain that 


m 


'S/Urt'S/h — A- 


|2*(si)-l, 


dx = o(l)||/i||. 


(8.79) 


dx = Q (8.80) 


(8.81) 


Hence, T'^(un) —)■ 0 in 77 ^(H). Similarly, we can prove that —)> 0 ir 


□ 


Corollary 8.4. liminf / ae„|u„|“|t;„|^ > 0. 

n^+oo 

Proof. By Lemma l872l we see that 


|K||^ + ||Un||^>4„>^0>0. 


Hence, we obtain that 


(8.82) 

(8.83) 


{Un,Vn) -h- ((^,0) in 

Ifliminf / |Mn|“|un,|^ = 0, by Lemma[02] we obtain that T'((u„) —0,■ 

n,-).+oo 

0 in H~^p). Since either Un 74 0 or Un, 0, it is easy to see that either 

lim x{un) > mx 

n—>-+00 

or 

lim > m^. 

n—>-+00 

By the assumption of lim inf / |M„|“|t;n|^ = 0 again, we have 

n_>+oo 

lim = lim ^'A(un) + > minjTOA, (8.84) 

n—>-+oo n^+00 

a contradiction to Lemma [8.11l Thereby this corollary is proved. □ 
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(8.85) 


Lemma 8.13. Assume that is a bounded sequence of S! such that 

lim = 0 

n—v+oo 

and 

lim inf Ce„ (()!)„,-^n) > 0, (8.86) 

n—>-+oo 

where the functionals J^(u,v) and c^{u,v) are defined in (18.111) and (18.101) . respec¬ 
tively. Then 

liminf > Co. (8.87) 

n—>-+oo 

Proof Since lim inf > 0, we see that {fm'fn) 7 ^ (0,0) in S). Without 

n—>-+oo 

loss of generality, we may assume that {4>n,'fn) ^ (0,0) for all n. Combining with 
the boundedness of {{fin, 'fin)}, we obtain that there exists some do,di > 0 such that 

0 < do < a{fin,fin) ■= \\fin\f + \\fin\f < di for all n. ( 8 . 88 ) 

We also claim that b{fin, fin) is bounded and away from 0, i.e., there exists da, d 4 > 0 
such that 

0 < da < b{fin,fin) ■■= + ^1^" l2* 

The right-hand inequality in ( 18.89b is trivial due to the Hardy-Sobolev inequality. Now, 
we only need to prove the existence of da. We proceed by contradiction. If b{fin,fin) 

0 up to a subsequence, then fin ^ 0,fin ^ 0 Strongly in j^rr). Recall¬ 

ing the boundedness of {{fin, fin)} again, by Proposition 12.11 and Proposition 122] we 
obtain that fin ^ 0,fin ^ 0 strongly in ^^^\fl,ao{x)dx). Noting that ae„{x) < 
ao(a;), then by the Holder inequality, it is easy to prove that 

^Sn{fin,fin) fi: Ct){fin,fin) ^ 0, (8.90) 

a contradiction to (18.86b and thereby (18.89b is proved. We also note that Ce„ {fin, fin) is 
bounded. Hence, up to a subsequence, we may assume that 

a{fin,fin) -)> a* > 0,b{fin,fin) b* > 0,Ce„{fin,fin) C* > 0. (8.91) 

Then by (lOST) . we see that 


a* -b* -2 *{s2)kc* =0. (8.92) 

On the other hand, for any n, by Lemma ISTl there exists a unique tyj > 0 such that 
Je„ {tnfin,tnfin) = 0 and tn is implicity given by the following equation 

a{fin,fin) - b{fin,fin)tn “ “2* {S 2 ) KCe^{fin, fin)tl = 0. (8.93) 


Since a{fin,fin) is bounded and lim inf (<)>„, ^/j^) > 0, it is easy to see that is 

n—>-+oo 

bounded. On the other hand, by the Hardy-Sobolev inequality again, we obtain that 




2 *(« 2 ) 


a{fin,fin) < Ci{a{fin,fin)) " ^ + <^2 (a((/'n , V’n)) "" ^,(8.94) 


for some positive constants Ci, C 2 independent of n. Then it is easy to see that at least 
one of the following holds; 
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2 *(ai) 


(i) ^a{(j)n,1pn) < Ci(a{(j>n,'4’n)) 

(n) ^a{cj)n,1pn) < C2{a{(l>n,1pn)) ^ 


Hence, we see that is bounded away from 0. Up to a subsequence if necessary, we 
assume that —?► f* > 0. Then we have that 


is bounded in 

tn ^ t* > 0, 

By dUB, we obtain that 

- 2*(s2)kc*(U)2*("=^ = 0, (U > 0). (8.95) 

It is easy to see that the algebraic equation a* = b*t^ ('<i)-2 _|_ (s 2)-2 ^ unique 

positive solution. Hence, by (18.92b and (18.95b . we obtain that f* = 1. Then recalling 
the boundedness of '*/’«)} again, we see that 


lim Jg 

n—¥-\-co 


Xt*^nX^n) = 0. 


lim lim $e„ fnV'n)- (8.96) 

n—v+oo n—>-+oo 

By the definition of f„, we see that {tn4>n,tn'4’n) € Me^- Hence, (tn'Pmtn'ipn) > 
Ce„. Then by ( 18.96b and Corollarv l8.2l we obtain that 

lim ^er,i4'n,ipn) > Hm C£„ = Cq. (8.97) 

n—>-+oo n—^-\-oo 

□ 


Lemma 8.14. Assume s\,S 2 G (0, 2),A,/r S (0,+oo),k > 0,a > l,/3 > 1 an^f 
a + f3 = 2 *(s2 ). Let e G (0, S2), then any solution {u, v) of the system (18.7b satisfies 


/ Kae(a;)|u|“|t;|^fia; = / Kae{x)\u\°‘\v\^ dx, 

JnnMi JQnMf 

where Bi is the unit ball of entered at zero. 

Proof Let 

G(x,u,v) = —-^-- V Kae(x)\u\^\v\ 

^ 2*(si) |a;h 2*(si) |a;h I I 1 


(8.98) 


(8.99) 


Noting that 


OXi 


Qeix) = 


-{S 2 -£)J^z^+J^x^ for |a;| < 1, 
^-(52 + g) |^|. 2 +^+e a^t for |x| > 1, 


( 8 . 100 ) 
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we have 


■ G^,{x,u,v) 


\-S, 

A 


1 


2*(si) 

kl"! 

' 2*(si) Ixpi 

jTp 


A 


1 U 


2*(si) 


' 2*(si) Ixpi 

|xp 


- is2-£) ^^pj2-e X^ if|a;|<l, 

- {s 2 + e) ^]T}J'Xe xi if|a;|>l. 


|2*(si) 


M \v 


2 *(si)_ 


-] dx 


Hence, by ( 17.29b . we have 

— 2(7V —S 2 ) [ Kae{x)\u\°'\v\^dx 
Jo, 

+ {N-2) [ {\yu\^+ \yv\^)dx 
Jn 

=2e ( Kae{x)\u\°'\v\^dx — 2£ f Kas{x)\u\°'\v\^dx. 

J f2n®i J oniB^ 

On the other hand, since (u, v) is a solution of the system ( 18.7b . we have 
(|Vup + |Vz;p)dx 


/n 


+ /r—-h 2*{s2)nae{x)\u\°'\v\^)dx. 


m ■ FI”! |a;| 

Since e > 0, by (18.102b and (18.103b . we obtain the result of (18.98b . 


( 8 . 101 ) 


( 8 . 102 ) 


(8.103) 

□ 


Proof ofTheorem l8.lt By Corollarv l8.31 we only need to prove that (uo, t^o) ^ (0,0). 
Now,we proceed by contradiction. We assume that (uo, t^o) = (0,0). By Corollary 
18.41 we have, up to a subsequence if necessary, that 


lim [ as„{x)\u„\°^\vn\^dx := T > 0. (8.104) 

On the other hand, by Corollary (04] we have 

/ ae^{x)\un\°'\vnfdx= ( ae^{x)\un\°'\vn\^dx for all n. (8.105) 

7nnBi innBf 


Hence, 


lim / ae„{x)\un\°‘\vnfdx = lim / aE„(a:)|u„|“|u„|'^da; = ^ > 0. 

JnnBi n->-+oo Jqpbc 2 


(8.106) 
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Let x{x) G C'^(R^) be a cut-off function such that x(a;) = 1 in Bi, x{x) = 0 in 
]R'^\Bi and take x{^) ^ C°°(R^) such that x(x) = 0 in Bi and x = 1 in R'^\B 2 . 

Denote 

:= x(x)Un(x), (l>2.n{x) ■= Xix)Un{x), 

\V'l.n(a;) := x{x)Vn{x), i> 2 ,n{x) := x{x)Vn{,x). 

Recalling that is a positive ground state solution of the system (I8.71 i with 

e = £n, then 

- (l)2,n,Vn - '01, n “ 0’2.n)) = 0 for all n. (8.108) 


when (uo,vo) = (0,0), by Rellich-Kondrachov compactness theorem, if 0 ^ fl, we 
have that —>• 0,u„ —>■ 0 strongly in j^rr) and (fi, uni¬ 

formly for all n. Hence, it is easy to prove that 

(Un - 01,n - (f>2,n,Vn “ 1pl,n “ '02,n) (0,0) StTOngly in (8.109) 

We also have that 

(^'^^(Un,Vn),(4>l,n,1pl,n)) = 0 for all 71. (8.110) 

Then by (uq, vq) = (0, 0) and Rellich-Kondrachov compactness theorem again, it is 
easy to see that 

lim Je„(0l,„,0l,n) = 0. (8.111) 

n—>-+oo 

We also obtain that 

liminf Ce„(0i,„,0i,„) = lim / a£„(x)|u„|“|?;„|^dx = - > 0. (8.112) 

n-)-|-oo n-)-|-oo 2 

Hence, by Lemma [8.131 we have 

lim $£„(0i,„,0i,n) > Co. (8.113) 

n—>-+oo 

Similarly, we can prove that 

lim 3>£„(02,„,02,n) > Co. (8.114) 

n—>-+oo 

By (uo, Vo) = (0,0) and the Rellich-Kondrachov compact theorem again, we have that 
Ce„ = lim $^^(un,Vn)= lim [$e„ (01,„, 01,„) + 4'e„ (02,n, 02,n)] • (8.115) 

n^+oo n^+oo 

Then by ( 18.1131 ).( 18.114b and Corollarv l8.2l we obtain that 


co>2co, (8.116) 

a contradict to the fact of that co > 0. Hence, (mo,uo) ^ (0,0) and the proof is 
completed. 
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